Android Things makes developing connected embedded devices easy by providing the same Android development tools, best-in-class Android framework, and Google APIs that make developers successful on mobile. Android Things extends the core Android framework with additional APIs provided by the Things Support Library. These APIs allow apps to integrate with new types of hardware not found on mobile devices.

The Google Assistant SDK lets you add voice control, natural language understanding, and Google's smarts to your devices. Your device captures an utterance (a spoken audio request, such as "What's on my calendar?"), sends it to the Google Assistant, and receives a spoken audio response in addition to the raw text of the utterance.

What you will build

In this codelab, you're going to use Android Things and the Google Assistant SDK to build your own custom Google Assistant device.

It will use a button to trigger recording from two Microelectromechanical (MEMS) microphones and play back the Assistant answer on an external speaker.

What you'll learn

What you'll need

Update Android SDK

Before you begin building apps for Android Things, you must:

Flash Android Things

If you have not already installed Android Things on your development board, follow the official image flashing instructions.

Assemble the hardware

Follow the assembly guide for the AIYProjects Voice Kit and make sure to use the Android Things SD card flashed in the previous step.

Connect to the device

  1. Verify that Android is running on the device. The Android Things Launcher shows information about the board, including the IP address, if you attach the Raspberry Pi to a monitor.
  2. Connect to this IP address using the adb tool:
$ adb connect <ip-address>
connected to <ip-address>:5555

Get the sample

Click the following link to download the sample for this codelab:

Download source code

...or you can clone the GitHub repository from the command line:

$ git clone https://github.com/googlecodelabs/androidthings-googleassistant.git

Unpack the downloaded zip file.

Configure the credentials

  1. Enable the following activity controls in the Google Account you plan to use with the Assistant:
  1. Make sure you check the box that says "Include Chrome browsing history and activity from websites and apps that use Google services"
  1. In the Cloud Platform Console, go to the Projects page. Select an existing project or create a new project.
  2. Enable the Google Assistant API on the project you selected (see the Terms of Service).
  3. Click Enable.
  4. Create an OAuth Client ID
  5. Click Other (select Other UI and User data if prompted) and give the client ID a name.
  6. On the OAuth consent screen tab, give the product a name (don't use "Google" in the name) and a description.
  7. Click Save.
  8. Click Create. A dialog box appears that shows you a client ID and secret. (No need to remember or save this, just close the dialog.)
  9. Click ⬇ (at the far right of screen) for the client ID to download the client secret JSON file (client_secret_NNNN.json or client_id.json).
  10. Open a terminal and follow the instructions to configure a new Python virtual environment and install the google-assistant-library.
  11. In your virtual environment, run the command:
# Install the authorization tool.
(env) $ pip install google-auth-oauthlib[tool]
  1. Navigate to your top-level project directory.
  2. Use the google-oauthlib-tool command line tool to grant the permission to use the Assistant API to your application and create a new credentials.json file in your app resource directory.
(env) $ cd <project-directory-name>
# Run the tool.
(env) $ google-oauthlib-tool --client-secrets path/to/client_secret_NNNN.json \
                       --credentials shared/src/main/res/raw/credentials.json \
                       --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
                       --save --headless

You will see a URL. Copy the URL and paste it into a browser. After you authorize your account, a code will appear in your browser, such as "4/XXXX". Copy and paste this code into the terminal.

It should then display: OAuth credentials initialized.

Run the sample

  1. Open Android Studio and select Import project.
  2. Select the androidthings-googleassistant directory from the extracted zip file location.
  3. Open the step1-start-here module in Android Studio.

Understanding the starter project

This project contains all of the code you need to have a fully functioning Assistant.

  1. Deploy the app to the device by selecting Run → Run 'step1-start-here' from the menu.
  1. Select the Android Monitor tab (typically at the bottom of the window) to see the logcat output inside Android Studio.

The project has successfully launched on the device if you can see the following startup message in the log:

... D AssistantActivity: Starting Assistant demo

Send a query

"What time is it?"

"It's one thirty."

Taking a look under the hood

There are a few key components to make this project work.

VoiceHat

The VoiceHat board uses the I2S protocol to read data from the microphones and write audio data to the speaker. This is handled automatically by the framework.

From your activity, you can use the AudioTrack and AudioRecord classes to interact with audio, just as if a mobile app was using the built-in microphone and speaker. With your driver registered, one can re-use source code and libraries designed for mobile without major changes in your activity.

Google Assistant gRPC API

The AssistantActivity class has multiple methods which make calls to the Google Assistant by streaming user voice data from the microphones. Each button press triggers a gRPC call through the Assistant SDK.

Local audio data is streamed in chunks to the Assistant, with each chunk wrapped in a ConverseRequest. As the audio requests are processed, the activity receives a ConverseResponse for various events. Each response may contain any of the following elements:

The Google Assistant API allows you to control the volume of the Assistant device thru voice with queries like:

"Turn the volume up"

"Turn the volume down"

"Set the volume to 4"

If you try those queries with the starter project, you will notice that the Assistant doesn't understand them yet. You must provide information about the current volume of the device before the Assistant can update it.

You can update the sample to include the current volume percentage of the device in your request, then update the volume of the AudioTrack when your app receives the result of the voice query.

Modify the AssistantActivity to:

  1. Add a new mVolumePercentage field to the AssistantActivity class.
private static int mVolumePercentage = 100;
  1. Update the AudioOutConfig to include the new volume parameter. This will be used to establish the current volume, allowing you to change it later.
ConverseConfig.Builder converseConfigBuilder = ConverseConfig.newBuilder()
                    .setAudioInConfig(AudioInConfig.newBuilder()
                            .setEncoding(ENCODING_INPUT)
                            .setSampleRateHertz(SAMPLE_RATE)
                            .build())
                    .setAudioOutConfig(AudioOutConfig.newBuilder()
                            .setEncoding(ENCODING_OUTPUT)
                            .setSampleRateHertz(SAMPLE_RATE)
                            .setVolumePercentage(mVolumePercentage)
                            .build());
  1. In the mAssistantResponseObserver.onNext(ConverseResponse value) method, handle volume percentage change from the ConverseResult of incoming ConverseResponse messages.
  2. Use the AudioTrack.setVolume method to update the volume of the assistant playback accordingly. The volume must be scaled to be in proportion to the AudioTrack limits.
 private StreamObserver<ConverseResponse> mAssistantResponseObserver =
            new StreamObserver<ConverseResponse>() {
        @Override
        public void onNext(ConverseResponse value) {
            switch (value.getConverseResponseCase()) {
                case RESULT:
                    // ...
                    if (value.getResult().getVolumePercentage() != 0) {
                        mVolumePercentage = value.getResult().getVolumePercentage();
                        Log.i(TAG, "assistant volume changed: " + mVolumePercentage);
                        float newVolume = mAudioTrack.getMaxVolume() * mVolumePercentage / 100.0f;
                        mAudioTrack.setVolume(newVolume);
                    }
                    // ...
                // ...
            }
        }
        // ...
    };

Try some volume queries again, they should now modify the volume of the playback of the Assistant:

"Turn the volume to 2"

You should see "assistant volume changed: 20" in logcat. There will be no verbal response.

"What sounds does a horse make?"

The Assistant answer should play back at a very low volume.

"Turn the volume to maximum"

You should see "assistant volume changed: 100" in logcat.

"What sounds does a horse make?"

The Assistant answer should play back at very loud volume.

Some Assistant Apps keep track of state, allowing the user to start and maintain a conversation with the Assistant.

User: "Talk to Number Genie"

Assistant device: "..., This is Number Genie: guess a number between 0 and 100"

User: "42"

Assistant device: "Higher"

If you try those queries with the sample currently, you will notice that the Assistant loses the context of the conversation between each turn of the dialog. The Assistant assumes every interaction is a new voice query because the app does not pass ConversationState into the next ConverseRequest.

You can update the sample to get the current state from ConverseResult and send the ConversationState in ConverseConfig.

Modify the AssistantActivity class.

  1. Add a new mConversationState field to the AssistantActivity class.
private ByteString mConversationState = null;
  1. In the mAssistantResponseObserver.onNext(ConverseResponse value) method, obtain the conversation state from the ConverseResult of incoming ConverseResponse messages.
case RESULT:
    // ...
    mConversationState = value.getResult().getConversationState();
  1. In the mStartAssistantRequest runnable, include the new conversation state before sending it with the outgoing ConverseRequest.
mAssistantRequestObserver = mAssistantService.converse(mAssistantResponseObserver);
// ...
ConverseConfig.Builder converseConfigBuilder =
ConverseConfig.newBuilder()
    .setAudioInConfig(ASSISTANT_AUDIO_REQUEST_CONFIG)
    .setAudioOutConfig(ASSISTANT_AUDIO_RESPONSE_CONFIG);
if (mConversationState != null) {
    converseConfigBuilder.setConverseState(
            ConverseState.newBuilder()
                    .setConversationState(mConversationState)
                    .build());
}
mAssistantRequestObserver.onNext(
        ConverseRequest.newBuilder()
                .setConfig(converseConfigBuilder.build())
                .build()
);

Invoke an Assistant app like Number Genie and observe that the state of your conversation is kept after each dialog turn.

User: "Ok, talk to Number Genie"

Assistant device: "..., This is Number Genie: guess a number between 0 and 100"

User: "42"

Assistant device: "Lower"

Congratulations! You've successfully integrated Google Assistant into your own device using Android Things.

Here are some ideas you can implement to go deeper.

Extend your assistant

Use Actions on Google and Firebase Cloud Messaging to extend your assistant with additional functionality.

What we've covered