Android Things makes developing connected embedded devices easy by providing the same Android development tools, best-in-class Android framework, and Google APIs that make developers successful on mobile. With the TensorFlow inference library for Android, developers can easily integrate TensorFlow and machine learning into their apps on Android Things.

TensorFlow™ is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. TensorFlow has become a popular framework for training machine learning models and using those models to solve problems.

What you'll build

In this codelab, you will use the TensorFlow inference library for Android to build a device that captures images from the device camera and locally classifies them against a pre-trained ImageNet model.

What you'll learn

What you'll need

Update Android SDK

Before you begin building apps for Things, you must:

Flash Android Things

If you have not already installed Android Things on your development board, follow the official image flashing instructions for your board:

Assemble the hardware

  1. Install the Rainbow HAT on top of your developer board.
  2. Connect the camera module to the connector marked CAMERA on your board.

Connect to the device

Verify that your development computer is properly connected to your device using the adb tool:

$ adb devices
List of devices attached
1b2f21d4e1fe0129        device

The expansion connector on the development board exposes Peripheral I/O signals for application use. The Rainbow HAT sits on top of the expansion connector, providing a variety of inputs and outputs for developers to interact with.

The peripherals on the Rainbow HAT used in this codelab are connected to the following signals. These are also listed on the back of the Rainbow HAT:

Peripheral Device

Raspberry Pi 3

i.MX7D

'C' Button

BCM16

GPIO_39

Camera Module

The developer boards are equipped with a Camera Serial Interface (CSI) connector to integrate supported camera modules with Android Things. The CSI bus is a high-speed, dedicated interface for capturing camera data. Supported camera modules connected to the developer board are accessed using the standard Android Camera APIs.

Click the following link to download the starter project for this codelab:

Download source code

...or you can clone the GitHub repository from the command line:

$ git clone https://github.com/googlecodelabs/androidthings-imageclassifier

Open the imageclassifier-start project in Android Studio and run it. Once the app launches on the device, look for a "READY" line in the Android Logcat output:

Android Logcat

... D/ImageClassifierActivity: READY

If you tap the 'C' button on the HAT, you should see a "RESULTS: I don't know what I see." line being printed on Android Logcat. Nothing else should happen at this point. In the next steps you will be adding Artificial Intelligence (TensorFlow) to actually recognize the photo, and camera capturing.

About the project

The starter project contains the following code:

Our current starter project doesn't do anything with the image. If you look at the doRecognize() method, it always return an empty array, meaning that nothing was recognized. Let's wire an Artificial Intelligence engine here, so that the Android Things device can actually recognize what it sees.

Add the TensorFlow Library

TensorFlow is an open-source library for machine learning and deep neural network created by Google. It is available as a Gradle dependency on JCenter:

Add the library dependency to your app-level build.gradle file.

build.gradle

dependencies {
    provided 'com.google.android.things:androidthings:0.4-devpreview'
    compile 'com.google.android.things.contrib:driver-rainbowhat:0.3'
    compile 'org.tensorflow:tensorflow-android:1.2.0-preview'
}

Add the TensorFlow calls

Open ImageClassifierActivity.java and make the following changes:

  1. Add code to initialize a TensorFlow model (the actual deep neural network) and the labels that maps the neural network results to names/categories recognized:

ImageClassifierActivity.java

import org.tensorflow.contrib.android.TensorFlowInferenceInterface;
 
public class ImageClassifierActivity extends Activity {
    ...
 
    private String[] labels;
    private TensorFlowInferenceInterface inferenceInterface;
 
    private void initClassifier() {
        this.inferenceInterface = new TensorFlowInferenceInterface(
                getAssets(), Helper.MODEL_FILE);
        this.labels = Helper.readLabels(this);
    }
 
    private void destroyClassifier() {
        inferenceInterface.close();
    } 

    ...
}
  1. Now let's implement the doRecognize() method. This method is called whenever the users requests an image classification. It takes the image as a parameter and runs the onPhotoRecognitionReady callback method with an array of Strings describing what was recognized in the input image (for example, ["beer bottle", "water bottle"]):

ImageClassifierActivity.java

public class ImageClassifierActivity extends Activity {
    ...
    private void doRecognize(Bitmap image) {
        float[] pixels = Helper.getPixels(image);
 
        // Feed the pixels of the image into the
        // TensorFlow Neural Network
        inferenceInterface.feed(Helper.INPUT_NAME, pixels,
                Helper.NETWORK_STRUCTURE);
 
        // Run the TensorFlow Neural Network with the provided input
        inferenceInterface.run(Helper.OUTPUT_NAMES);
 
        // Extract the output from the neural network back
        // into an array of confidence per category
        float[] outputs = new float[Helper.NUM_CLASSES];
        inferenceInterface.fetch(Helper.OUTPUT_NAME, outputs);
 
        // Send to the callback the results with the highest
        // confidence and their labels
        onPhotoRecognitionReady(Helper.getBestResults(outputs, labels));
    }
 
    ...
}

That's it. Now, if you run your app, you should see on Android Logcat that it is recognizing something when you press the "C" button.

Android Logcat

... D/ImageClassifierActivity: Running photo recognition
... D/ImageClassifierActivity: Using sample photo in res/drawable/sampledog_224x224.png
... D/ImageClassifierActivity: RESULTS: I see a Tibetan terrier or maybe a Bouvier des Flandres

As you can see from the logs, by default we use a static image of a dog:

This is "Proto", an adorable dog from a fellow Googler. I know you want to spend some time looking at him, so take your time...

... Ok, that's enough. Let's move on to the next step, so that we can add code to capture the image from the camera instead of using Proto's photo (I know, I know).

Now we will add code to fetch an image from the board's camera, so you will be able to point the camera at objects and test the object recognition more properly.

Add Camera permission

To access the camera, you will need proper permission.

Add the permission to your app's manifest file.

AndroidManifest.xml

<!-- // ADD CAMERA SUPPORT -->
<uses-permission android:name="android.permission.CAMERA"/>

Update ImageClassifierActivity to use the camera

Let's add the code that will initialize and shutdown the camera and take photos. Open ImageClassifierActivity and make the following changes:

  1. Add the variables that will hold the camera-related objects:

ImageClassifierActivity.java

...
import android.media.ImageReader;
import android.os.Handler;

public class ImageClassifierActivity extends Activity {

    ...
    // ADD CAMERA SUPPORT
    private CameraHandler mCameraHandler;
    private ImagePreprocessor mImagePreprocessor;

    ...
}
  1. Now update the initCamera() method so that it initializes the ImagePreprocessor and CameraHandler objects.
  2. Implement an OnImageAvailableListener to be called when an image from the camera is ready. The listener should invoke ImagePreprocessor.preprocessImage() and pass the processed Bitmap to onPhotoReady(), which will reroute it to the image recognition method you implemented in the previous section.
  3. Pass the new listener to CameraHandler.initializeCamera():

ImageClassifierActivity.java

/**
 * Initialize the camera that will be used to capture images.
 */
private void initCamera() {
    // ADD CAMERA SUPPORT
    mImagePreprocessor = new ImagePreprocessor();
    mCameraHandler = CameraHandler.getInstance();
    Handler threadLooper = new Handler(getMainLooper());

    mCameraHandler.initializeCamera(this, threadLooper,
            new ImageReader.OnImageAvailableListener() {
        @Override
        public void onImageAvailable(ImageReader imageReader) {
            Bitmap bitmap = mImagePreprocessor.preprocessImage(imageReader.acquireNextImage());
            onPhotoReady(bitmap);
        }
    });
}
  1. Finally, implement closeCamera() and loadPhoto() with code that closes the camera and triggers the CameraHandler:

ImageClassifierActivity.java

/**
 * Clean up resources used by the camera.
 */
private void closeCamera() {
    // ADD CAMERA SUPPORT
    mCameraHandler.shutDown();
}

/**
 * Load the image that will be used in the classification process.
 * When done, the method {@link #onPhotoReady(Bitmap)} must be called with the image.
 */
private void loadPhoto() {
    // ADD CAMERA SUPPORT
    // Replace the existing code in loadPhoto() by the following:
    mCameraHandler.takePicture();
}

That's it. Install the app on your board (reboot it if you are using Android Studio older than 3.0) and run. When you press the "C" button you should see several messages on logcat, something like this:

Android Logcat

... D/ImageClassifierActivity: Running photo recognition
...
... D/CameraHandler: Capture request created.
...
... D/ImageHelper: Saving 224x224 bitmap to /storage/emulated/0/Pictures/tensorflow_preview.png.
...
... D/ImageClassifierActivity: RESULTS: I see a window shade
... D/CameraHandler: CaptureSession closed
...

Congratulations! You've successfully built an image classifier using TensorFlow and Android Things! Here are some things you can do to go deeper.

What we've covered