Android Things makes developing connected embedded devices easy by providing the same Android development tools, best-in-class Android framework, and Google APIs that make developers successful on mobile. Android Things extends the core Android framework with additional APIs provided by the Things Support Library. These APIs allow apps to integrate with new types of hardware not found on mobile devices.

The Google Assistant SDK lets you add voice control, natural language understanding and Google's smarts to your devices. Your device captures an utterance (a spoken audio request, such as "What's on my calendar?"), sends it to the Google Assistant, and receives a spoken audio response in addition to the raw text of the utterance.

What you will build

In this codelab, you're going to use Android Things and the Google Assistant SDK to build your own custom Google Assistant device.

It will use a button to trigger recording from two MEMS microphones and playback the Assistant answer on an external speaker.

What you'll learn

What you'll need

Update Android SDK

Before you begin building apps for Android Things, you must:

Flash Android Things

If you have not already installed Android Things on your development board, follow the official image flashing instructions.

Make sure you enable I2S by changing the following line in config.txt on the first SD card partition.

# comment or remove this line:
# dtoverlay=pwm-2chan-with-clk,pin=18,func=2,pin2=13,func2=4
#
# uncomment or add this line:
dtoverlay=generic-i2s

Assemble the hardware

Follow the assembly guide for the AIYProjects Voice Kit and make sure to use the Android Things SD card flashed in the previous step.

Connect to the device

  1. Verify that Android is running on the device. The Android Things Launcher shows information about the board, including the IP address.
  2. Connect to this IP address using the adb tool:
$ adb connect <ip-address>
connected to <ip-address>:5555

Get the sample

Click the following link to download the sample for this codelab:

Download source code

...or you can clone the GitHub repository from the command line:

$ git clone https://github.com/androidthings/sample-googleassistant
  1. Unpack the downloaded zip file.
  2. Open Android Studio and select Import project.
  3. Select the sample-googleassistant directory from the extracted zip file location.

Configure the credentials

  1. Enable the following activity controls in the Google Account you plan to use with the Assistant:
  1. In the Cloud Platform Console, go to the Projects page. Select an existing project or create a new project.
  2. Enable the Google Assistant API on the project you selected (see the Terms of Service).
  3. Click Enable.
  4. Create an OAuth Client ID
  5. Click Other (select Other UI and User data if prompted) and give the client ID a name.
  6. On the OAuth consent screen tab, give the product a name (don't use "Google" in the name) and a description.
  7. Click Save.
  8. Click Create. A dialog box appears that shows you a client ID and secret. (No need to remember or save this, just close the dialog.)
  9. Click ⬇ (at the far right of screen) for the client ID to download the client secret JSON file (client_secret_NNNN.json or client_id.json).
  10. Open a terminal and install the google-auth-lib command line tool:
$ pip install google-auth-oauthlib[tool] --user
  1. Navigate to your top-level project directory.
  2. Use the google-oauthlib-tool command line tool to grant the permission to use the Assistant API to your application and create a new credentials.json file in your app resource directory.
$ cd <project-directory-name>
$ google-oauthlib-tool --client-secrets path/to/client_secret_NNNN.json \
                       --credentials app/src/main/res/raw/credentials.json \
                       --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
                       --save

Replace path/to/client_secret_NNNN.json with the path of the JSON file you downloaded in step 10.

Connect to the Raspberry Pi.

Make sure you're connected to the Raspberry Pi over adb.

$ adb connect Android.local
connected to Android.local:5555

Run the sample

Do the following steps from within Android Studio:

  1. Deploy the app to the device by selecting Run → Run 'app' from the menu.
  1. Select the Android Monitor tab (typically at the bottom of the window) to see the logcat output inside Android Studio.

The project has successfully launched on the device if you can see the following startup message in the log:

... D AssistantActivity: Starting Assistant demo

Send a query

"What time is it?"

"It's eleven past fourty two."

Audio User Driver

The VoiceHatDriver class shows how to:

private class AudioInputUserDriver extends AudioInputDriver {
   // implement driver.
}
// register driver.
mAudioInputDriver = new AudioInputUserDriver();
UserDriverManager.getManager().registerAudioInputDriver(
    mAudioInputDriver,
    mAudioFormat,
    AudioDeviceInfo.TYPE_BUILTIN_MIC,
    BUFFER_SIZE
);
private class AudioOutputUserDriver extends AudioOutputDriver {
   // implement driver.
}
// register driver.
mAudioOutputDriver = new AudioOutputUserDriver();
UserDriverManager.getManager().registerAudioOutputDriver(
    mAudioOutputDriver,
    mAudioFormat,
    AudioDeviceInfo.TYPE_BUILTIN_SPEAKER,
    BUFFER_SIZE
);
I2sDevice mDevice;
// open I2S device.
mDevice = pioService.openI2sDevice(i2sBus, audioFormat);
// read samples.
mDevice.read(byteBuffer, n);
// write samples.
mDevice.write(byteBuffer, n);

Google Assistant gRPC API

The AssistantActivity class shows how to:

// call Converse streaming RPC method.
mAssistantRequestObserver = mAssistantService.converse(mAssistantResponseObserver);
// send audio config.
mAssistantRequestObserver.onNext(
    ConverseRequest.newBuilder().setConfig(
        ConverseConfig.newBuilder()
            .setAudioInConfig(ASSISTANT_AUDIO_REQUEST_CONFIG)
            .setAudioOutConfig(ASSISTANT_AUDIO_RESPONSE_CONFIG)
            .build()
    ).build()
);
// send audio payloads.
mAssistantRequestObserver.onNext(
    ConverseRequest.newBuilder()
         .setAudioIn(ByteString.copyFrom(audioData))
         .build()
);
// receive ConverseResponse stream.
StreamObserver<ConverseResponse> mAssistantResponseObserver = 
    new StreamObserver<ConverseResponse>() {
        @Override
        public void onNext(ConverseResponse value) {
            switch (value.getConverseResponseCase()) {
                case EVENT_TYPE:
                    // handle events.
                    break;
                case RESULT:
                    // handle query result.
                case AUDIO_OUT:
                    // handle audio data of the assistant's answer.
                    break;
                // ...
            }
        }
    };

Congratulations! You've successfully built your own Google Assistant using Android Things. Here are some things you can do to go deeper.

Add volume control

The Google Assistant API allows you to control the volume of the Assistant device thru voice with queries like:

"Turn the volume up"

"Turn the volume down"

"Set the volume to 4"

If you try those queries with the sample, you will notice that the Assistant doesn't understand them yet. Because it doesn't currently know about the current volume of our device, it doesn't really know how to update it.

You can update the sample to send the current volume percentage of the device in ConverseConfig , and update the volume of the AudioTrack according to the ConverseResult of those voice query.

Modify the AssistantActivity.java to:

private int mVolumePercentage = 100;
// ...
.setAudioOutConfig(AudioOutConfig.newBuilder()
    .setEncoding(ENCODING_OUTPUT)
    .setSampleRateHertz(SAMPLE_RATE)
    .setVolumePercentage(mVolumePercentage).build())
case RESULT:
    // ...
    if (value.getResult().getVolumePercentage() != 0) {
        mVolumePercentage = value.getResult().getVolumePercentage();
        Log.i(TAG, "assistant volume changed: " + mVolumePercentage);
        mAudioTrack.setVolume(mAudioTrack.getMaxVolume() * mVolumePercentage / 100.f);
    }
    break;

"Turn the volume to 2"

You should see "assistant volume changed: 20" in logcat (w/ no assistant answer).

"What sounds does a horse makes?"

The assistant answer should play back at a very low volume.

"Turn the volume to maximum"

You should see "assistant volume changed: 100" in logcat.

"What sounds does a horse makes?"

The assistant answer should play back at very loud volume.

Add support for conversational agent

Some Assistant app supports Conversation Actions, allowing the user to have a long running stateful dialog with the Assistant device:

User: "Ok, talk to Number Genie"

Assistant device: "..., This is Number Genie: guess a number between 0 and 100"

User: "42"

Assistant device: "Higher"

If you try those queries with the sample, you will notice that the Assistant loose the context of the conversation between each turn of the dialog. Because we don't send any ConversationState the assistant assume every interaction is a new voice query.

You can update the sample to get the current conversation state from ConverseResult and send the ConversationState in ConverseConfig.

Modify the AssistantActivity class to:

private ByteString mConversationState = null;
case RESULT:
    // ...
    mConversationState = value.getResult().getConversationState();
mAssistantRequestObserver = mAssistantService.converse(mAssistantResponseObserver);
// ...
ConverseConfig.Builder converseConfigBuilder =
ConverseConfig.newBuilder()
    .setAudioInConfig(ASSISTANT_AUDIO_REQUEST_CONFIG)
    .setAudioOutConfig(AudioOutConfig.newBuilder()
                            .setEncoding(ENCODING_OUTPUT)
                            .setSampleRateHertz(SAMPLE_RATE)
                            .setVolumePercentage(mVolumePercentage)
                            .build());
if (mConversationState != null) {
    converseConfigBuilder.setConverseState(
            ConverseState.newBuilder()
                    .setConversationState(mConversationState)
                    .build());
}
mAssistantRequestObserver.onNext(
        ConverseRequest.newBuilder()
                .setConfig(converseConfigBuilder.build())
                .build()
);

User: "Ok, talk to Number Genie"

Assistant device: "..., This is Number Genie: guess a number between 0 and 100"

User: "42"

Assistant device: "Lower"

Extend your assistant

Use Actions on Google and Firebase Messaging to extend your assistant with new functionality.

What we've covered