This codelab will teach you how to integrate the Local Home SDK with your existing smart home Action to support local execution.

What is local execution?

Smart home integrations extend the functionality of the Google Assistant to control connected devices. To build a smart home Action, developers provide a cloud webhook endpoint capable of handling smart home intents. For instance, when a user says "Hey Google, turn on the lights,'' the Google Assistant sends this command to your cloud fulfillment to update the state of the device.

The Local Home SDK enhances your smart home integration by adding a local execution path to route smart home intents directly to a Home device. This enhances reliability and reduces latency in processing user commands.

Using the SDK, you write and deploy a local execution app, using TypeScript or JavaScript, to identify devices and execute commands locally on any Google Home smart speaker or Nest smart display. Your app communicates directly with your existing smart devices over the local area network (LAN), using existing standard protocols, to fulfill user commands.

What you'll build

In this codelab, you will deploy a pre-built smart home integration with Firebase, then implement local execution by applying a scan configuration in the Actions console and building a local app using TypeScript (TS) which sends commands to a virtual washer device (written in Node.js) over HTTP.

We strongly recommend that you familiarize yourself with the topics covered in the smart home documentation before starting this codelab. For more details about building a smart home Action, see Create a smart home Action.

What you'll learn

What you'll need

Enabling Activity controls

Enable the following Activity controls in the Google Account you plan to use with the Assistant:

Using the Actions on Google console

  1. Go to the Actions on Google Developer Console.
  2. Click on Add Project, enter a name for the project, and click CREATE PROJECT.

Select the Smart Home App

On the Overview screen in the Actions console, select Smart home.

Choose the Smart home experience card, and you will then be directed to your project console.

Install the Firebase Command Line Interface

The Firebase Command Line Interface (CLI) will allow you to serve your web apps locally and deploy your web app to Firebase hosting.

To install the CLI, run the following npm command from the terminal:

npm install -g firebase-tools

To verify that the CLI has been installed correctly, run:

firebase --version

Authorize the Firebase CLI with your Google account by running:

firebase login

Enable the HomeGraph API

The HomeGraph API enables the storage and querying of devices and their states within a user's Home Graph. To use this API, you must first open the Google Cloud console and enable the HomeGraph API.

In the Google Cloud console, make sure to select the project that matches your Actions <project-id>. Then, in the API Library screen for the HomeGraph API, click Enable.

Now that you've set up your development environment, let's deploy the starter project to verify everything is configured properly.

Get the source code

Click the following link to download the sample for this codelab on your development machine:

Download source code

...or you can clone the GitHub repository from the command line:

git clone https://github.com/googlecodelabs/smarthome-local.git

About the project

The starter project contains the following subdirectories:

The provided cloud fulfillment includes the following functions in index.js:

Connect to Firebase

Navigate to the app-start directory then set up the Firebase CLI with your Actions Project:

cd app-start
firebase use <project-id>

Deploy to Firebase

Navigate to the functions folder and install all the necessary dependencies using npm.

cd functions
npm install

Now that you have installed the dependencies and configured your project, you are ready to run the app for the first time.

firebase deploy

This is the console output you should see:

...

✔ Deploy complete!

Project Console: https://console.firebase.google.com/project/<project-id>/overview
Hosting URL: https://<project-id>.firebaseapp.com

This command deploys a web app, along with several Cloud Functions for Firebase.

Open the Hosting URL in your browser (https://<project-id>.firebaseapp.com) to view the web app. You will see the following interface:

This web UI represents a third-party platform to view or modify device states. To begin populating your database with device information, click UPDATE. You won't see any changes on the page, but the current state of your washer will be stored in the database.

Now it's time to connect the cloud service you've deployed to the Google Assistant using the Actions console.

Configure your project in the Actions console

Under Overview > Build your Action, select Add Action(s). Enter the URL for your cloud function that provides fulfillment for the smart home intents and click Save.

https://us-central1-<project-id>.cloudfunctions.net/smarthome

On the Develop > Invocation tab, add a Display Name for your Action, and click Save. This name will appear in the Google Home app.

To enable Account linking, select the Develop > Account linking option in the left navigation. Use these account linking settings:

Client ID

ABC123

Client secret

DEF456

Authorization URL

https://us-central1-<project-id>.cloudfunctions.net/fakeauth

Token URL

https://us-central1-<project-id>.cloudfunctions.net/faketoken

Click Save to save your account linking configuration, then click Test to enable testing on your project.

You will be redirected to the Simulator. Verify that testing has been enabled for your project by moving your mouse over the Testing on Device ( ) icon.

Link to the Google Assistant

In order to test your smart home Action, you need to link your project with a Google account. This enables testing through Google Assistant surfaces and the Google Home app that are signed in to the same account.

  1. On your phone, open the Google Assistant settings. Note that you should be logged in as the same account as in the console.
  2. Navigate to Google Assistant > Settings > Home Control (under Assistant).
  3. Select the plus (+) icon in the bottom right corner
  4. You should see your test app with the [test] prefix and the display name you set.
  5. Select that item. The Google Assistant will then authenticate with your service and send a SYNC request, asking your service to provide a list of devices for the user.

Open the Google Home app and verify that you can see your washer device.

Verify that you can control the washer using voice commands in the Google Home app. You should also see the device state change in the frontend web UI of your cloud fulfillment.

Now that we have a working implementation in the cloud, let's explore adding the local execution path.

To support local execution, we need to add a new per-device field called otherDeviceIds to the cloud SYNC response. This field indicates that a device can be controlled locally and provides the identifier that the local execution app must return during device discovery.

Add the otherDeviceIds field to the SYNC response as shown below:

functions/index.js

app.onSync((body) => {
  return {
    requestId: body.requestId,
    payload: {
      agentUserId: '123',
      devices: [{
        id: 'washer',
        type: 'action.devices.types.WASHER',
        traits: [ ... ],
        name: { ... },
        deviceInfo: { ... },
        willReportState: true,
        attributes: {
          pausable: true,
        },
        otherDeviceIds: [{
          deviceId: 'deviceid123',
        }],
      }],
    },
  };
});

We also need a way for devices to report local state changes to the cloud. When your app receives local execution commands from Google, your cloud service is not directly notified of the command. Instead, the device is responsible for publishing those updates to keep the state synchronized.

Add a new updateState function to the cloud fulfillment that accepts the HTTP request and updates Firebase with the parameters received:

functions/index.js

/**
 * Update the current state of the washer device
 */
exports.updateState = functions.https.onRequest((request, response) => {
  firebaseRef.child('washer').update({
    OnOff: {
      on: request.body.on,
    },
    StartStop: {
      isPaused: request.body.isPaused,
      isRunning: request.body.isRunning,
    }
  });
  return response.status(200).end();
});

Deploy the updated project to Firebase:

firebase deploy --only functions

After deployment completes, navigate to the web UI and click the refresh icon in the toolbar. This triggers a Request Sync operation so the Google Assistant receives the updated SYNC response data.

In this section, we are going to add the necessary configuration options for local execution to our smart home Action. During development, we will publish the local execution app to Firebase Hosting, where the Google Home device can access and download it.

In the Actions console, select Test > On device testing. Enter the following URL into the text field (inserting your own project id), then click Save:

https://<project-id>.firebaseapp.com/local-home/index.html

Next, we need to define how the Google Home device should discover the local smart devices. The local execution framework supports several protocols device discovery, including mDNS, UPnP and UDP broadcast. In this codelab, we will use UDP broadcast to discover our virtual device.

Select Develop > Actions and find the Configure local home SDK section. Add the following attributes under Device scan configuration (click Add scan configuration to include additional fields):

Field

Description

Suggested value

UDP discovery address

UDP broadcast address

255.255.255.255

UDP discovery port out

Port where Google Home sends

the UDP broadcast

3311

UDP discovery port in

Port where Google Home listens

for a response

3312

UDP discovery packet

UDP broadcast data payload

48656c6c6f4c6f63616c486f6d6553444b

Finally, click Save at the top of the window to publish your changes.

You will develop your local execution app in TypeScript using the Local Home SDK typings package. Let's look at the skeleton provided in the starter project:

local/index.ts

/// <reference types="@google/local-home-sdk" />

import App = smarthome.App;
import Constants = smarthome.Constants;
import DataFlow = smarthome.DataFlow;
import Execute = smarthome.Execute;
import Intents = smarthome.Intents;
import IntentFlow = smarthome.IntentFlow;

...

class LocalExecutionApp {

  constructor(private readonly app: App) { }

  identifyHandler(request: IntentFlow.IdentifyRequest):
      Promise<IntentFlow.IdentifyResponse> {
    // TODO: Implement device identification
  }

  executeHandler(request: IntentFlow.ExecuteRequest):
      Promise<IntentFlow.ExecuteResponse> {
    // TODO: Implement local execution
  }

  ...
}

const localHomeSdk = new App('1.0.0');
const localApp = new LocalExecutionApp(localHomeSdk);
localHomeSdk
  .onIdentify(localApp.identifyHandler.bind(localApp))
  .onExecute(localApp.executeHandler.bind(localApp))
  .listen()
  .then(() => console.log('Ready'))
  .catch((e: Error) => console.error(e));

The core component of local execution is the smarthome.App class. The starter project attaches handlers for the IDENTIFY and EXECUTE intents, then calls the listen() method to inform Local Home SDK that the app is ready.

Add the IDENTIFY Handler

The Local Home SDK triggers your IDENTIFY handler when the Google Home device discovers unverified devices on the local network, based on the scan configuration provided in the Actions console.

When a matching device is discovered, the identifyHandler is invoked with the resulting scan data. In our application, scanning takes place using a UDP broadcast and the scan data provided to the IDENTIFY handler includes the response payload sent by the local device.

The handler returns an IdentifyResponse containing a unique identifier for the local device. Add the following code to your identifyHandler() to process the UDP response coming from the local device and determine the appropriate local device id:

local/index.ts

identifyHandler(request: IntentFlow.IdentifyRequest):
    Promise<IntentFlow.IdentifyResponse> {

  const scanData = request.inputs[0].payload.device.udpScanData;
  if (!scanData) {
    const err = new IntentFlow.HandlerError(request.requestId,
        'invalid_request', 'Invalid scan data');
    return Promise.reject(err);
  }

  // In this codelab, the scan data contains only local device id.
  const localDeviceId = Buffer.from(scanData, 'hex');

  const response: IntentFlow.IdentifyResponse = {
    intent: Intents.IDENTIFY,
    requestId: request.requestId,
    payload: {
      device: {
        id: 'washer',
        verificationId: localDeviceId.toString(),
      }
    }
  };

  return Promise.resolve(response);
}

Note that the verificationId must match one of the otherDeviceIds in your SYNC response; this flags the device as available for local execution in the user's Home Graph. After Google successfully finds a match, that device is verified and ready for local execution.

Add the EXECUTE Handler

The Local Home SDK triggers your EXECUTE handler when a command is sent to a device that supports local execution. The content of the local intent is equivalent to the EXECUTE intent sent to your cloud fulfillment, so the logic for processing the intent locally will be similar to how you handle it in the cloud.

The app can use TCP/UDP sockets or HTTP(S) requests to communicate with local devices. In this codelab, HTTP is the protocol used to control the virtual device. The port number is defined in index.ts as the SERVER_PORT variable.

Add the following code to your executeHandler() to process incoming commands and send them to the local device over HTTP:

local/index.ts

executeHandler(request: IntentFlow.ExecuteRequest):
    Promise<IntentFlow.ExecuteResponse> {

  const command = request.inputs[0].payload.commands[0];
  const execution = command.execution[0];
  const response = new Execute.Response.Builder()
    .setRequestId(request.requestId);

  const promises: Array<Promise<void>> = command.devices.map((device) => {

    // Convert execution params to a string for the local device
    const params = execution.params as IWasherParams;
    const payload = this.getDataForCommand(execution.command, params);

    // Create a command to send over the local network
    const radioCommand = new DataFlow.HttpRequestData();
    radioCommand.requestId = request.requestId;
    radioCommand.deviceId = device.id;
    radioCommand.data = JSON.stringify(payload);
    radioCommand.dataType = 'application/json';
    radioCommand.port = SERVER_PORT;
    radioCommand.method = Constants.HttpOperation.POST;
    radioCommand.isSecure = false;

    return this.app.getDeviceManager()
      .send(radioCommand)
      .then(() => {
        const state = {online: true};
        response.setSuccessState(device.id, Object.assign(state, params));
        console.log(`Command successfully sent to ${device.id}`);
      })
      .catch((e: IntentFlow.HandlerError) => {
        e.errorCode = e.errorCode || 'invalid_request';
        response.setErrorState(device.id, e.errorCode);
        console.error('An error occurred sending the command', e.errorCode);
      });
  });

  return Promise.all(promises)
    .then(() => {
      return response.build();
    })
    .catch((e) => {
      const err = new IntentFlow.HandlerError(request.requestId,
          'invalid_request', e.message);
      return Promise.reject(err);
    });
}

Compile the TypeScript app

Navigate to the local/ directory and run the following commands to download the TypeScript compiler and compile the app:

cd local
npm install
npm run build

This compiles the index.ts (TypeScript) source and places the following contents into the public/local-home/ directory:

Deploy the test project

Deploy the updated project files to Firebase Hosting so we can access them from the Home device.

firebase deploy --only hosting

Now it's time to test the communication between your local execution app and a smart home device! The codelab starter project includes a Virtual Smart Washer, written in Node.js, which simulates a smart washer that can be controlled locally.

Configure the device

We need to configure the virtual device to use the same UDP parameters we applied to the scan configuration for device discovery in the Actions console. Additionally, we need to tell the virtual device which local device ID to report and a URL where it should send a Report State event when the device state changes.

Parameter

Suggested value

deviceId

deviceid123

discoveryPortOut

3311

discoveryPacket

HelloLocalHomeSDK

reportStateUrl

https://us-central1-<project-id>.cloudfunctions.net/updateState

Start the device

Navigate to the virtual-device/ directory and run the device script, passing the configuration parameters as arguments:

cd virtual-device
npm install
npm start -- \
  --deviceId=deviceid123 \
  --discoveryPortOut=3311 --discoveryPacket=HelloLocalHomeSDK \
  --reportStateUrl="https://us-central1-<project-id>.cloudfunctions.net/updateState"

Verify that the device script is running with the expected parameters:

(...): UDP Server listening on 3311
(...): Device listening on port 3388
(...): Report State successful

In this section, we will verify that the Google Home device can properly scan, identify, and send commands to our virtual device over the local network. We can use Chrome DevTools to connect to the Google Home device, view the console logs, and debug the TypeScript app.

Connect Chrome DevTools

To connect the debugger to your local execution app, follow the steps below:

  1. Make sure your Google Home device is linked to a user that has permission to the Actions console project.
  2. Reboot your Google Home device. This step enables the device to get the URL of your HTML as well as the scan config you put in the Actions console.
  3. Launch Chrome on your development machine.
  4. Open a new Chrome tab and enter chrome://inspect in the address field to launch Chrome inspector.

You should see a list of devices on the page, and your app URL should be listed under the name of the Google Home device you are using.

Launch the inspector

Click the blue inspect link under your app URL to launch Chrome DevTools. Select the Console tab and verify you can see the content of IDENTIFY intent printed by your TypeScript app.

This output means your local execution app has successfully discovered and identified the virtual device.

Test local execution

Send commands to your device using the touch controls in the Google Home app or through voice commands to the Google Home device, such as:

"Turn on my washer"

"Start my washer"

"Stop my washer"

This utterance should trigger an EXECUTE intent sent to your TypeScript app.

Verify that you can see the virtual smart washer device change state change locally with each command.

...
{
  "isRunning": true
}
***** The washer is RUNNING *****
...
{
  "isRunning": false
}
***** The washer is STOPPED *****

Congratulations! You have successfully integrated local execution into a smart home Actions with the Local Home SDK. Here are some additional things you can try:

What we've covered