Android Things makes developing connected embedded devices easy by providing the same Android development tools, best-in-class Android framework, and Google APIs that make developers successful on mobile. Android Things extends the core Android framework with additional APIs provided by the Things Support Library. These APIs allow apps to integrate with new types of hardware not found on mobile devices.

The Google Assistant SDK lets you add voice control, natural language understanding, and Google's smarts to your devices. Your device captures an utterance (a spoken audio request, such as "What's on my calendar?"), sends it to the Google Assistant, and receives a spoken audio response in addition to the raw text of the utterance.

What you will build

In this codelab, you're going to use Android Things and the Google Assistant SDK to build your own custom Google Assistant device.

It will use a button to trigger a microphone and play back the Assistant answer on an external speaker.

What you'll learn

What you'll need

Update Android SDK

Before you begin building apps for Android Things, you must:

Flash Android Things

If you have not already installed Android Things on your development board, follow the official image flashing instructions.

Assemble the hardware

If you are using the AIYProjects Voice Kit, follow the assembly guide for the and make sure to use the Android Things SD card flashed in the previous step.

Otherwise, you can get started by connecting the board to power.

Connect to the device

  1. Verify that Android is running on the device. The Android Things Launcher shows information about the board, including the IP address, if you attach the Raspberry Pi to a monitor.
  2. Connect to this IP address using the adb tool:
$ adb connect <ip-address>
connected to <ip-address>:5555

Get the sample

Click the following link to download the sample for this codelab on your development machine:

Download source code

...or you can clone the GitHub repository from the command line:

$ git clone https://github.com/googlecodelabs/androidthings-googleassistant.git

Unpack the downloaded zip file.

Configure the credentials

  1. Enable the following Activity controls in the Google Account you plan to use with the Assistant:
  1. Open the Actions Console and either select an existing project or create a new project.
  1. Select the Device registration tab (under ADVANCED OPTIONS) from the left navbar.
  2. Click the REGISTER MODEL button

  1. Click ⬇Download credentials.json for the client ID to download the client secret JSON file (credentials.json).
  1. Enable the Google Assistant API in the Cloud Console
  2. Open a terminal on your development machine and follow the instructions to configure a new Python virtual environment.
$ python3 -m venv env
$ source env/bin/activate
(env) $ pip install --upgrade pip setuptools wheel
(env) $ pip install --upgrade google-auth-oauthlib[tool]
  1. Navigate to your top-level project directory.
  2. Use the google-oauthlib-tool command line tool to grant permission to use the Assistant API to your application and create a new credentials.json file in your app resource directory.
(env) $ cd <project-directory-name>
# Run the tool.
(env) $ google-oauthlib-tool --client-secrets path/to/credentials.json \
                       --credentials shared/src/main/res/raw/credentials.json \
                       --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
                       --save

This will open a browser and ask you to authorize the application to make request to the assistant on your behalf.

It should then display: credentials saved: shared/src/main/res/raw/credentials.json.

Run the sample

  1. Open Android Studio and select Import project.
  2. Select the androidthings-googleassistant directory from the extracted zip file location.
  3. Open the step1-start-here module in Android Studio.

Understanding the starter project

This project contains all of the code you need to have a fully functioning Assistant.

Set the DeviceConfig

Before you run the project, you will need to update the app with the device model and instance ids that you registered. Open up shared/src/main/java/com/example/androidthings/assistant/shared/MyDevice.java. Update the values of MODEL_ID and INSTANCE_ID.

The MODEL_ID should be the model id created using the Actions Console. The INSTANCE_ID should be a unique identifier for each device.

public class MyDevice {
    public static final String MODEL_ID = "model-id-from-the-action-console";
    public static final String INSTANCE_ID = "some-identifier-unique-to-your-project";
}
  1. Deploy the app to the device by selecting Run → Run 'step1-start-here' from the menu.
  1. Select the Logcat tab (typically at the bottom of the window) to see the logcat output inside Android Studio.

The project has successfully launched on the device if you can see the following startup message in the log:

... D AssistantActivity: Starting Assistant demo

Send a query

"What time is it?"

"It's one thirty."

Taking a look under the hood

There are a few key components to make this project work.

VoiceHat

The VoiceHat board uses the I2S protocol to read data from the microphones and write audio data to the speaker. This is handled automatically by the framework.

From your activity, you can use the AudioTrack and AudioRecord classes to interact with audio, just as if a mobile app was using the built-in microphone and speaker. With your driver registered, one can re-use source code and libraries designed for mobile without major changes in your activity.

Google Assistant gRPC API

The AssistantActivity class has multiple methods which make calls to the Google Assistant by streaming user voice data from the microphones. Each button press triggers a gRPC call through the Assistant SDK.

Local audio data is streamed in chunks to the Assistant, with each chunk wrapped in a AssistRequest. As the audio requests are processed, the activity receives a AssistResponse for various events. Each response may contain any of the following elements:

The Google Assistant API allows you to control the volume of the Assistant device thru voice with queries like:

"Turn the volume up"

"Turn the volume down"

"Set the volume to 4"

If you try those queries with the starter project, you will notice that the Assistant doesn't understand them yet. You must provide information about the current volume of the device before the Assistant can update it.

You can update the sample to include the current volume percentage of the device in your request, then update the volume of the AudioTrack when your app receives the result of the voice query.

Modify the AssistantActivity to:

  1. Add a new mVolumePercentage field to the AssistantActivity class.
private static int mVolumePercentage = 100;
  1. Update the AssistConfig to replace ASSISTANT_AUDIO_RESPONSE_CONFIG with a new AudioOutConfig containing the new volume parameter. This will be used to establish the current volume, allowing you to change it later.
AssistConfig.Builder converseConfigBuilder = AssistConfig.newBuilder()
                    .setAudioInConfig(ASSISTANT_AUDIO_REQUEST_CONFIG)
                    .setAudioOutConfig(AudioOutConfig.newBuilder()
                            .setEncoding(ENCODING_OUTPUT)
                            .setSampleRateHertz(SAMPLE_RATE)
                            .setVolumePercentage(mVolumePercentage)
                            .build())
                    .setDeviceConfig(DeviceConfig.newBuilder()
                            .setDeviceModelId(MyDevice.MODEL_ID)
                            .setDeviceId(MyDevice.INSTANCE_ID)
                            .build());
  1. In the mAssistantResponseObserver.onNext(AssistResponse value) method, handle volume percentage change from the AssistResult of incoming AssistResponse messages.
  2. Use the AudioTrack.setVolume method to update the volume of the assistant playback accordingly. The volume must be scaled to be in proportion to the AudioTrack limits.
 private StreamObserver<AssistResponse> mAssistantResponseObserver =
            new StreamObserver<AssistResponse>() {
        @Override
        public void onNext(AssistResponse value) {
            // ...
            if (value.getDialogStateOut() != null) {
                int volume = value.getDialogStateOut().getVolumePercentage();
                if (volume > 0) {
                    mVolumePercentage = volume;
                    Log.i(TAG, "assistant volume changed: " + mVolumePercentage);
                    mAudioTrack.setVolume(AudioTrack.getMaxVolume() * mVolumePercentage / 100.0f);
                }
            }
            // ...

        }
        // ...
    };

Try some volume queries again, they should now modify the volume of the playback of the Assistant:

"Turn the volume to 2"

You should see "assistant volume changed: 20" in logcat. There will be no verbal response.

"What sounds does a horse make?"

The Assistant answer should play back at a very low volume.

"Turn the volume to maximum"

You should see "assistant volume changed: 100" in logcat.

"What sounds does a horse make?"

The Assistant answer should play back at very loud volume.

In addition to talking to the Google Assistant, you may want your device to perform certain actions like turning on and off an LED or setting its brightness. To do this, you can use built-in device actions. When you say a command that your device can support, it will receive a JSON payload that will allow you to handle the query directly.

The schema is the same as the EXECUTE payload for Smart Home devices.

To start with Device actions, you will need to register your device with the types of actions it can support. You have already registered your device in the Actions Console. Return to your project and open the device model in the Device registration section. Click the icon in Supported traits.

Select the OnOff trait and then press SAVE.

Create a new device instance for your updated model using the INSTANCE_ID that you added earlier in: shared/src/main/java/com/example/androidthings/assistant/shared/MyDevice.java.

You can find your PROJECT_ID in the URL of the Actions console https://console.actions.google.com/project/project-id/..., or in the project settings behind the ⚙ button.

(env) $ google-oauthlib-tool --client-secrets path/to/credentials.json \
                       --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
                       --save
(env) $ pip install google-assistant-sdk
(env) $ googlesamples-assistant-devicetool --project-id PROJECT_ID list --model
...
(env) $ googlesamples-assistant-devicetool --project-id PROJECT_ID register-device \
--model MODEL_ID --device DEVICE_INSTANCE_ID --client-type SERVICE
...

When a Device action command is said, the AssistResponse will include a Device action as a serialized JSON payload.

@Override
public void onNext(AssistResponse value) {
    // ...
    if (value.getDeviceAction() != null &&
            !value.getDeviceAction().getDeviceRequestJson().isEmpty()) {
        // Iterate through JSON object
        try {
            JSONObject deviceAction = 
                new JSONObject(value.getDeviceAction().getDeviceRequestJson());
            JSONArray inputs = deviceAction.getJSONArray("inputs");
            for (int i = 0; i < inputs.length(); i++) {
                if (inputs.getJSONObject(i).getString("intent")
                        .equals("action.devices.EXECUTE")) {
                    JSONArray commands = inputs.getJSONObject(i)
                        .getJSONObject("payload")
                        .getJSONArray("commands");
                    for (int j = 0; j < commands.length(); j++) {
                        JSONArray execution = commands.getJSONObject(j)
                            .getJSONArray("execution");
                        for (int k = 0; k < execution.length(); k++) {         
                            String command = execution.getJSONObject(k)
                                  .getString("command");
                            JSONObject params = execution.getJSONObject(k)
                                  .optJSONObject("params");
                            handleDeviceAction(command, params);
                        }
                    }
                }
            }
        } catch (JSONException | IOException e) {
            e.printStackTrace();
        }
    }
    // ...
}
         

Create a new method called handleDeviceAction which will take the command and parameters and execute the request.

public void handleDeviceAction(String command, JSONObject params)
       throws JSONException, IOException {
    if (command.equals("action.devices.commands.OnOff")) {
        mLed.setValue(params.getBoolean("on"));
    }
}    

User: "Turn on"

Assistant device: <Turns on LED>

You can easily change the LED actuated by the device actions to something else, by opening a new Gpio device and referring to the table below:

RED LED (above Button A)

GPIO2_IO02

GREEN LED (above Button B)

GPIO2_IO00

BLUE LED (above Button C)

GPIO2_IO05

In addition to talking to the Google Assistant, you may want your device to perform certain actions that are not supported by built-in traits, which may only make sense for your device. To do this, you can use custom device actions. These actions can be defined with a custom grammar which may include parameters. When you say a command that your device can support, it will receive a JSON payload that will allow you to handle the query directly. The schema is the same as the EXECUTE payload for Smart Home devices.

To start with custom device actions, you will need to create an action package which will define all of the query patterns and parameters. In this codelab, you will add a blink action. You will have to specify the ability for it to blink a particular number of times, as well as the frequency which will be defined as a custom type.

Copy the snippet below into a file called actions.json.

{
  "manifest": {
    "displayName": "Blinky light",
    "invocationName": "Blinky light",
    "category": "PRODUCTIVITY"
  },
  "actions": [
    {
      "name": "com.example.actions.BlinkLight",
      "availability": {
        "deviceClasses": [
          {
            "assistantSdkDevice": {}
          }
        ]
      },
      "intent": {
        "name": "com.example.intents.BlinkLight",
        "parameters": [
          {
            "name": "number",
            "type": "SchemaOrg_Number"
          },
          {
            "name": "speed",
            "type": "Speed"
          }
        ],
        "trigger": {
          "queryPatterns": [
            "blink ($Speed:speed)? $SchemaOrg_Number:number times",
            "blink $SchemaOrg_Number:number times ($Speed:speed)?"
          ]
        }
      },
      "fulfillment": {
        "staticFulfillment": {
          "templatedResponse": {
            "items": [
              {
                "simpleResponse": {
                  "textToSpeech": "Blinking $speed.raw $number times"
                }
              },
              {
                "deviceExecution": {
                  "command": "com.example.commands.BlinkLight",
                  "params": {
                    "speed": "$speed",
                    "number": "$number"
                  }
                }
              }
            ]
          }
        }
      }
    }
  ],
  "types": [
    {
      "name": "$Speed",
      "entities": [
        {
          "key": "slowly",
          "synonyms": [
            "slow"
          ]
        },
        {
          "key": "normally",
          "synonyms": [
            "regular"
          ]
        },
        {
          "key": "quickly",
          "synonyms": [
            "fast",
            "quick"
          ]
        }
      ]
    }
  ]
}
 

There are two query patterns defined:

These patterns include two parameters. One of these is $SchemaOrg_Number:number, which represents the number of times to blink the light. The other is a custom type called $Speed:speed which is optional. We define this type as having three entities: slowly, normally, and quickly. There are also synonyms which will match the entities.

When one of the query patterns is said, the fulfillment will be executed. This includes a custom defined phrase and TTS audio. A callback will be sent to the device.

Download the gactions tool, and use it to start testing this action package, replacing project_id below with the id for your project.

(env) $ gactions test --action_package actions.json --project project_id

In the snippet above, you have put this action package in testing mode. Now your device will receive the command "com.example.commands.BlinkLight". You will need to handle this with a new check in the handleDeviceAction method.

In order to not block on the main thread, you can use a Handler and the postDelayed method.

public void handleDeviceAction(String command, JSONObject params)
       throws JSONException, IOException {
    if (command.equals("action.devices.commands.OnOff")) {
        mLed.setValue(params.getBoolean("on"));
    } else if (command.equals("com.example.commands.BlinkLight")) {
        int delay = 1000;
        int blinkCount = params.getInt("number");
        String speed = params.getString("speed");
        if (speed.equals("slowly")) {
            delay = 2000;
        } else if (speed.equals("quickly")) {
            delay = 500;
        }
        for (int i = 0; i < blinkCount*2; i++) {
            new Handler(Looper.getMainLooper()).postDelayed(() -> {
                try {
                    mLed.setValue(!mLed.getValue());
                } catch (IOException e) {
                    throw new RuntimeException(e);
                }
            }, i * delay);
        }
    }
}

User: "Blink fast 4 times"

Assistant device: <Turns on and off LED>

Congratulations! You've successfully integrated Google Assistant into your own device using Android Things.

Here are some ideas you can implement to go deeper.

Extend your assistant

Use Actions on Google and Firebase Cloud Messaging to extend your assistant with additional functionality.

Add more supported traits to your device.

What we've covered