Internet of Things (IoT) developers can build devices that can be remotely controlled from the Google Assistant through direct voice commands using the Smart Home integration. This occurs through a cloud-to-cloud integration between the Google Assistant and your server.

Smart Home apps rely on Home Graph, a database that stores and provides contextual data about the home and its devices. For example, Home Graph can store the concept of a living room that contains multiple types of devices (a thermostat, lamp, fan, and vacuum) from different manufacturers. This information is passed to the Google Assistant in order to execute user requests based on the appropriate context.

What you will build

In this codelab, you're going to create your own cloud integration and connect the Google Assistant to a smart washing machine.

What you'll learn

What you'll need

Enabling Activity controls

Enable the following Activity controls in the Google Account you plan to use with the Assistant:

Using the Actions on Google console

  1. Go to the Actions on Google Developer Console.
  2. Click on Add Project, enter a name for the project, and click CREATE PROJECT.

Select the Smart Home App

On the Overview screen in the Actions console, click Home control and then Smart home.

Install the Firebase Command Line Interface

The Firebase Command Line Interface (CLI) will allow you to serve your web apps locally and deploy your web app to Firebase hosting.

To install the CLI run the following npm command:

npm -g install firebase-tools

To verify that the CLI has been installed correctly open a console and run:

firebase --version

Make sure the Firebase version is above 3.3.0

Authorize the Firebase CLI by running:

firebase login

Get the sample

Click the following link to download the sample for this codelab on your development machine:

Download source code

...or you can clone the GitHub repository from the command line:

$ git clone https://github.com/googlecodelabs/smarthome-washer.git

Unpack the downloaded zip file.

Connect to Firebase

Make sure you are in the washer-start directory then set up the Firebase CLI to use your Firebase Project:

cd washer-start
firebase use --add

Then select your Project ID and follow the instructions.

Deploy to Firebase

Navigate to the functions folder inside of washer-start and run npm install.

cd functions
npm install

Now that you have installed the dependencies and configured your project, you are ready to run the app for the first time.

cd ../
firebase deploy

This is the console output you should see:

...

✔ Deploy complete!

Project Console: https://console.firebase.google.com/project/<project-id>/overview
Hosting URL: https://<project-id>.firebaseapp.com

The web app should now be served from your Hosting URL - which is of the form https://<project-id>.firebaseapp.com.

This command will also deploy several cloud functions for Firebase. You will now use these URLs in the Actions on Google console.

Configure your project in the Actions on Google console

Select Actions on the left-hand side and click ADD YOUR FIRST ACTION. Enter a URL for the backend server that will provide fulfillment for the Smart Home intents and then click DONE.

https://us-central1-<project-id>.cloudfunctions.net/smarthome

Then click DONE.

Select the Account Linking option in the sidebar. Select No for account creation, and make sure the linking type is OAuth and Authentication code.

Enter the following client information:

Client ID

ABC123

Client secret

DEF456

Authorization URL

https://us-central1-<project-id>.cloudfunctions.net/fakeauth

Token URL

https://us-central1-<project-id>.cloudfunctions.net/faketoken

Then select the SAVE button at the bottom to save this information and return to the Overview page.

Open the Simulator page and click the START TESTING button to finish your project setup.

Link to the Google Assistant

To finish setup, you will need to link to your Google account to your Smart Home cloud, which are the functions that you have deployed.

Now that your Smart Home account is connected to your Google Assistant, you can start adding devices and sending data. There are three intents that your server will need to handle in the smarthome function.

In order to dynamically get a list of devices, you can use Cloud Functions for Firebase. It will handle these three intents with responses that you can define. In order to host the state of your washer, you can use the Firebase Realtime Database.

Update SYNC response

Open functions/index.js. This contains the code to respond to requests from the Google Assistant. First, we will need to handle a SYNC intent by responding with our washer. You will see right now that it will always give the same basic response.

Implement the full JSON in the array to represent your washer.

app.onSync(body => {
  return {
    requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
    payload: {
      agentUserId: '123', 
      devices: [{
        id: 'washer',
        type: 'action.devices.types.WASHER',
        traits: [
          'action.devices.traits.OnOff',
          'action.devices.traits.StartStop',
          'action.devices.traits.RunCycle'
        ],
        name: {
          defaultNames: ['My Washer'],
          name: 'Washer',
          nicknames: ['Washer']
        },
        deviceInfo: {
          manufacturer: 'Acme Co',
          model: 'acme-washer',
          hwVersion: '1.0',
          swVersion: '1.0.1'
        },
        attributes: {
          pausable: true
        }
     }]
    }
  };
});

Now deploy the updated function.

firebase deploy

Re-link your integration to the Google Assistant

In order to test your new SYNC response, you will need to unlink your integration and link to it again. Later, you will add the Request Sync feature so that you will be able to cause a SYNC request without having to unlink your account.

Initializing the database

Now you will have to handle the two other intents, to let the user know the current state of their washer, or control it. Let's add the EXECUTE intent.

Go to the Firebase console and select your project. Then, open the Database page and GET STARTED with using the Realtime Database. The database is organized to store the states for each device. The hierarchy is shown below.

To set this up, you can open the hosted website for your project <project-id>.firebaseapp.com and click on the UPDATE button. It will immediately set the values to their defaults.

Handling EXECUTE intent

In functions/index.js edit the EXECUTE handler as below:

app.onExecute((body) => {
  const {requestId} = body;
  const payload = {
    commands: [{
      ids: [],
      status: 'SUCCESS',
      states: {
        online: true,
      },
    }],
  };
  for (const input of body.inputs) {
    for (const command of input.payload.commands) {
      for (const device of command.devices) {
        const deviceId = device.id;
        payload.commands[0].ids.push(deviceId);
        for (const execution of command.execution) {
          const execCommand = execution.command;
          const {params} = execution;
          switch (execCommand) {
            case 'action.devices.commands.OnOff':
              firebaseRef.child(deviceId).child('OnOff').update({
                on: params.on,
              });
              payload.commands[0].states.on = params.on;
              break;
            case 'action.devices.commands.StartStop':
              firebaseRef.child(deviceId).child('StartStop').update({
                isRunning: params.start,
              });
              payload.commands[0].states.isRunning = params.start;
              break;
            case 'action.devices.commands.PauseUnpause':
              firebaseRef.child(deviceId).child('StartStop').update({
                isPaused: params.pause,
              });
              payload.commands[0].states.isPaused = params.pause;
              break;
          }
        }
      }
    }
  }
  return {
    requestId: requestId,
    payload: payload,
  };
});

In the implementation above, you iterate through each command, update the value in Firebase, then respond with a payload stating the current state of your device.

Now deploy the updated function.

firebase deploy

Now you can see the value change when you give a voice command. You can use your phone to give these commands.

"Turn on my washer"

"Pause my washer"

"Stop my washer"

In order to support interrogative questions, like "Is my washer on?" you will need to implement a QUERY intent. You do this in the next step.

Handling QUERY intent

A QUERY intent will include a set of devices. For each device, you should respond with its current state.

In functions/index.js edit the QUERY handler as below:

app.onQuery((body) => {
  const {requestId} = body;
  const payload = {
    devices: {},
  };
  const queryPromises = [];
  for (const input of body.inputs) {
    for (const device of input.payload.devices) {
      const deviceId = device.id;
      queryPromises.push(queryDevice(deviceId)
        .then((data) => {
          // Add response to device payload
          payload.devices[deviceId] = data;
        }
        ));
    }
  }
  // Wait for all promises to resolve
  return Promise.all(queryPromises).then((values) => ({
    requestId: requestId,
    payload: payload,
  })
  );
});

Now you can see the current state of your washer by asking questions.

"Is my washer on?"

"Is my washer running?"

Now that you have implemented all three intents, you can add additional features to your washer.

Modes and toggles allow you to control a specific component of your device with a name defined by the developer. In this codelab, your washer has a mode to define the size of the laundry load: small or large.

To start, uncomment the section of index.html to show the modes:

<div id='demo-washer-modes-main'>
    <label>Washer Mode</label>
    <br>
    <label id='demo-washer-modes-small' class='mdl-radio mdl-js-radio mdl-js-ripple-effect' for='demo-washer-modes-small-in'>
      <input checked class='mdl-radio__button' id='demo-washer-modes-small-in' name='load' type='radio'
               value='on'>
      <span class='mdl-radio__label'>Small</span>
    </label>
    <label id='demo-washer-modes-large' class='mdl-radio mdl-js-radio mdl-js-ripple-effect' for='demo-washer-modes-large-in'>
      <input class='mdl-radio__button' id='demo-washer-modes-large-in' name='load' type='radio' value='off'>
      <span class='mdl-radio__label'>Large</span>
    </label>
    <br>
    <br>
</div>

When you press the UPDATE button, the mode will be stored in Firebase.

In your SYNC response, you will need to add information about this new mode. This will appear in the attributes object as shown in the response below.

app.onSync(body => {
  return {
    requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
    payload: {
      agentUserId: '123',
      devices: [{
        id: 'washer',
        type: 'action.devices.types.WASHER',
        traits: [
          'action.devices.traits.OnOff',
          'action.devices.traits.StartStop',
          'action.devices.traits.RunCycle',
          'action.devices.traits.Modes',
        ],
        name: {
          defaultNames: ['My Washer'],
          name: 'Washer',
          nicknames: ['Washer']
        },
        deviceInfo: {
          manufacturer: 'Acme Co',
          model: 'acme-washer',
          hwVersion: '1.0',
          swVersion: '1.0.1'
        },
        attributes: {
          pausable: true,
          availableModes: [{
              name: 'load',
              name_values: [{
                  name_synonym: ['load'],
                  lang: 'en'
                }],
              settings: [{
                  setting_name: 'small',
                  setting_values: [{
                      setting_synonym: ['small'],
                      lang: 'en'
                    }]
                  }, {
                  setting_name: 'large',
                  setting_values: [{
                      setting_synonym: ['large'],
                      lang: 'en'
                    }]
                }],
              ordered: true
            }]        
        }
    }]
    }
  };
});

In your EXECUTE intent, you will need to add the action.devices.commands.SetModes command, as shown below.

app.onExecute((body) => {
  const {requestId} = body;
  const payload = {
    commands: [{
      ids: [],
      status: 'SUCCESS',
      states: {
        online: true,
      },
    }],
  };
  for (const input of body.inputs) {
    for (const command of input.payload.commands) {
      for (const device of command.devices) {
        const deviceId = device.id;
        payload.commands[0].ids.push(deviceId);
        for (const execution of command.execution) {
          const execCommand = execution.command;
          const {params} = execution;
          switch (execCommand) {
            case 'action.devices.commands.OnOff':
              firebaseRef.child(deviceId).child('OnOff').update({
                on: params.on,
              });
              payload.commands[0].states.on = params.on;
              break;
            case 'action.devices.commands.StartStop':
              firebaseRef.child(deviceId).child('StartStop').update({
                isRunning: params.start,
              });
              payload.commands[0].states.isRunning = params.start;
              break;
            case 'action.devices.commands.PauseUnpause':
              firebaseRef.child(deviceId).child('StartStop').update({
                isPaused: params.pause,
              });
              payload.commands[0].states.isPaused = params.pause;
              break;
            case 'action.devices.commands.SetModes':
              firebaseRef.child(deviceId).child('Modes').update({
                load: params.updateModeSettings.load,
              });
              break;
          }
        }
      }
    }
  }
  return {
    requestId: requestId,
    payload: payload,
  };
});

Now you can give a command to set the mode of the washer.

"Set the washer to a large load"

Finally, you will need to update your QUERY response to respond to questions about the washer's current state. Add the updated changes to the queryFirebase and queryDevice functions to obtain the mode.

const queryFirebase = (deviceId) => firebaseRef.child(deviceId).once('value')
  .then((snapshot) => {
    const snapshotVal = snapshot.val();
    return {
      on: snapshotVal.OnOff.on,
      isPaused: snapshotVal.StartStop.isPaused,
      isRunning: snapshotVal.StartStop.isRunning,
      load: snapshotVal.Modes.load,
    };
  });

const queryDevice = (deviceId) =>    queryFirebase(deviceId).then((data) => ({
  on: data.on,
  isPaused: data.isPaused,
  isRunning: data.isRunning,
  currentRunCycle: [{
    currentCycle: 'rinse',
    nextCycle: 'spin',
    lang: 'en',
  }],
  currentTotalRemainingTime: 1212,
  currentCycleRemainingTime: 301,
  currentModeSettings: {
    load: data.load,
  },
}));

You will be able to ask questions about your washer such as:

"Is my washer small load?"

Toggles represent named aspects of a device that have a true/false state, such as whether the washer is in Turbo mode.

To start, uncomment the section of index.html to show the modes. When you press the UPDATE button, the mode will be stored in Firebase.

<div id='demo-washer-toggles-main'>
  <label id='demo-washer-toggles' class='mdl-switch mdl-js-switch mdl-js-ripple-effect' for='demo-washer-toggles-in'>
    <input type='checkbox' id='demo-washer-toggles-in' class='mdl-switch__input'>
    <span class='mdl-switch__label'>Is in Turbo</span>
  </label>
</div>

In your SYNC response, you will need to add information about this new mode. This will appear in the attributes object as shown in the response below.

app.onSync(body => {
  return {
    requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
    payload: {
      agentUserId: '123',
      devices: [{
        id: 'washer',
        type: 'action.devices.types.WASHER',
        traits: [
          'action.devices.traits.OnOff',
          'action.devices.traits.StartStop',
          'action.devices.traits.RunCycle',
          'action.devices.traits.Modes',
          'action.devices.traits.Toggles',
        ],
        name: {
          defaultNames: ['My Washer'],
          name: 'Washer',
          nicknames: ['Washer']
        },
        deviceInfo: {
          manufacturer: 'Acme Co',
          model: 'acme-washer',
          hwVersion: '1.0',
          swVersion: '1.0.1'
        },
        attributes: {
          pausable: true,
          availableModes: [{
              name: 'load',
              name_values: [{
                  name_synonym: ['load'],
                  lang: 'en'
                }],
              settings: [{
                  setting_name: 'small',
                  setting_values: [{
                      setting_synonym: ['small'],
                      lang: 'en'
                    }]
                  }, {
                  setting_name: 'large',
                  setting_values: [{
                      setting_synonym: ['large'],
                      lang: 'en'
                    }]
                }],
              ordered: true
            }],
          availableToggles: [{
              name: 'Turbo',
              name_values: [{
                  name_synonym: ['turbo'],
                  lang: 'en'
              }]
          }]
        }
    }]
    }
  };
});

In your EXECUTE intent, you will need to add the action.devices.commands.SetToggles command, as shown below.

app.onExecute((body) => {
  const {requestId} = body;
  const payload = {
    commands: [{
      ids: [],
      status: 'SUCCESS',
      states: {
        online: true,
      },
    }],
  };
  for (const input of body.inputs) {
    for (const command of input.payload.commands) {
      for (const device of command.devices) {
        const deviceId = device.id;
        payload.commands[0].ids.push(deviceId);
        for (const execution of command.execution) {
          const execCommand = execution.command;
          const {params} = execution;
          switch (execCommand) {
            case 'action.devices.commands.OnOff':
              firebaseRef.child(deviceId).child('OnOff').update({
                on: params.on,
              });
              payload.commands[0].states.on = params.on;
              break;
            case 'action.devices.commands.StartStop':
              firebaseRef.child(deviceId).child('StartStop').update({
                isRunning: params.start,
              });
              payload.commands[0].states.isRunning = params.start;
              break;
            case 'action.devices.commands.PauseUnpause':
              firebaseRef.child(deviceId).child('StartStop').update({
                isPaused: params.pause,
              });
              payload.commands[0].states.isPaused = params.pause;
              break;
            case 'action.devices.commands.SetModes':
              firebaseRef.child(deviceId).child('Modes').update({
                load: params.updateModeSettings.load,
              });
              break;
            case 'action.devices.commands.SetToggles':
              firebaseRef.child(deviceId).child('Toggles').update({
                Turbo: params.updateToggleSettings.Turbo,
              });
              break;
          }
        }
      }
    }
  }
  return {
    requestId: requestId,
    payload: payload,
  };
});


Now you can give a command to set the mode of the washer.

"Turn on turbo for the washer"

Finally, you will need to update your QUERY response to respond to questions about the washer's turbo mode. Add the updated changes to the queryFirebase and queryDevice functions to obtain the toggle state.

const queryFirebase = (deviceId) => firebaseRef.child(deviceId).once('value')
  .then((snapshot) => {
    const snapshotVal = snapshot.val();
    return {
      on: snapshotVal.OnOff.on,
      isPaused: snapshotVal.StartStop.isPaused,
      isRunning: snapshotVal.StartStop.isRunning,
      load: snapshotVal.Modes.load,
      turbo: snapshotVal.Toggles.Turbo,
    };
  });

const queryDevice = (deviceId) => queryFirebase(deviceId).then((data) => ({
  on: data.on,
  isPaused: data.isPaused,
  isRunning: data.isRunning,
  currentRunCycle: [{
    currentCycle: 'rinse',
    nextCycle: 'spin',
    lang: 'en',
  }],
  currentTotalRemainingTime: 1212,
  currentCycleRemainingTime: 301,
  currentModeSettings: {
    load: data.load,
  },
  currentToggleSettings: {
    Turbo: data.turbo,
  },
}));

You will be able to ask questions about your washer such as

"Is my washer in turbo mode?"

Now you have completely implemented your washer. You can control and get the current state of this washer. However, each time you added a new feature you had to unlink and relink to your integration.

A SYNC request has only reached your server during this linking step. By adding the Request Sync API, you will be able to trigger a SYNC request and allow a user's list of devices to be updated without unlinking and relinking their account.

The Request Sync API takes a user id as a parameter. This should be in the user id in your server that represents your user. In this codelab, the value is hardcoded to "123".

  1. In the Cloud Platform Console, go to the Projects page. Select the project that matches your Smart Home project id.
  2. Enable the HomeGraph API.
  3. Generate an API key. From the left navbar, select Credentials under APIs & Services. Click the Create Credentials button and select API key.
  4. Add the API key to the Smart Home constructor in functions/index.js
const app = smarthome({
  debug: true,
  key: '<api-key>'
});

In the frontend web UI, there is a refresh icon in the header. You can add a click listener to that in order to make your request sync call.

In main.js:

    this.requestSync = document.getElementById('request-sync');
    this.requestSync.addEventListener('click', () => {
      var xhttp = new XMLHttpRequest();
      xhttp.onreadystatechange = function() {
        if (this.readyState == 4 && this.status == 200) {
          console.log('Request SYNC success!');
        }
      };
      xhttp.open('POST', 'https://us-central1-<project-id>.cloudfunctions.net/requestsync', true);
      xhttp.send();
    });

Make sure to replace https://us-central1-<project-id>.cloudfunctions.net/requestsync with the equivalent function for your project.

Now when you click that icon, you will see a SYNC request in your server logs. Each time you update the device, add new devices, or remove this device, you will see it appear in your list of devices when you send a new request sync call.

The response will also appear in your logs, which you can view through Firebase.

With the Report State API, a Smart Home integration can proactively send the device's state to the home graph. This allows user queries to be completed faster and enables them to use rich UIs on their phones or smart displays in order to know the status of all their devices.

When you click the UPDATE button to change the washer's state, you can add an additional step which will report this to the home graph.

Create JWT

Before you can write our function, you will need to make sure our data is sent securely. This is done through JWT (JSON web tokens). In this codelab you will be using the googleapis npm dependency which will help facilitate the creation of this token.

Go to the Google Cloud Console for your project. Select Credentials in the APIs & Services section. Click the Create Credentials button and select Service account key. For Role, select Project -> Editor.

After creating your service account, you will download a JSON file. Save this file under the functions folder in your project with the name key.json.

Add API call

This can be done using a Firebase database trigger. When a certain write event happens, it can automatically report the state.

In functions/index.js:

const app = smarthome({
  debug: true,
  key: '<api-key>',
  jwt: require('./key.json'),
});
exports.reportstate = functions.database.ref('{deviceId}').onWrite((event) => {
  console.info('Firebase write event triggered this cloud function');
  const snapshotVal = event.data.val();

  const postData = {
    requestId: 'ff36a3cc', /* Any unique ID */
    agentUserId: '123', /* Hardcoded user ID */
    payload: {
      devices: {
        states: {
          /* Report the current state of our washer */
          [event.params.deviceId]: {
            on: snapshotVal.OnOff.on,
            isPaused: snapshotVal.StartStop.isPaused,
            isRunning: snapshotVal.StartStop.isRunning,
          },
        },
      },
    },
  };

  return app.reportState(postData)
    .then((data) => {
      console.log('Report state came back');
      console.info(data);
    });
});

Congratulations! You've successfully integrated Google Assistant into your own device using Smart Home.

Here are some ideas you can implement to go deeper.

If you're a device manufacturer, after you integrate your device with the Google Assistant, you can submit your action for review. Your integration with go through a certification process and be submitted.

What we've covered