Recognize Flowers with TensorFlow Lite on Android

1. Introduction

Note: This codelab requires a physical device to test

TensorFlow is a multipurpose machine learning framework. TensorFlow can be used anywhere from training huge models across clusters in the cloud, to running models locally on an embedded system like your phone.

This codelab uses TensorFlow Lite to run an image recognition model on an Android device.

Install Android Studio 4.1 or above

If you don't have it installed already, go download and install AndroidStudio 4.1 or above while you are training your TensorFlow Lite model.

What you'll learn

• How to train your own custom image classifier using TensorFlow Lite Model Maker.
• How to use Android Studio to import the TensorFlow Lite model to integrate the custom model in an Android app using CameraX.
• How to use GPU on your phone to accelerate your model.

What you will build

A simple camera app that runs a TensorFlow image recognition program to identify flowers.

License: Free to use

2. Train a flower recognizer using Colab

Before kicking off the model training, start downloading and installing Android Studio 4.1 or above.

Open the Colab which shows how to train a classifier with Keras to recognize flowers using TensorFlow Lite transfer learning.

Be sure that you:

1. downloaded the trained model (model.tflite), and
2. renamed the file FlowerModel.tflite

before continuing.

Some browsers will block the downloads:

You may need to give colab permission to download the files:

3. Set up the working directory

Clone the Git repository

The following command will clone the Git repository containing the files required for this codelab:

git clone https://github.com/hoitab/TFLClassify.git

Next, go to the directory you just cloned the repository. This is where you will be working on for the rest of this codelab:

cd TFLClassify

4. Set up the Android skeleton app

Install Android Studio 4.1 or above

If you don't have it installed already, go install AndroidStudio 4.1 or above.

Open the project with Android Studio

Open a project with Android Studio by taking the following steps:

1. Open Android Studio . After it loads select "Open an Existing project" from this popup:

2. In the file selector, choose TFLClassify/build.gradle from your working directory.

The project contains two modules, start and finish. If you get stuck, refer to the finish module for a hint.

3. You will get a "Gradle Sync" popup, the first time you open the project, asking about using gradle wrapper. Click "OK".

4. Enable developer model and USB Debugging on your phone if you have not already. This is a one-time set up. Follow these instructions.
5. Once both your project and your phone is ready, you can run it on a real device by selecting TFL_Classify.start and press the run button on the toolbar:

6. Now allow the Tensorflow Demo to access your camera:

7. You will see the following screen on your phone with random numbers taking the place of where real results will be displayed.

5. Add TensorFlow Lite to the Android app

1. Select the start module in the project explorer on the left hand side:

2. Right-click on the start module or click on File, then New > Other > TensorFlow Lite Model

3. Select the model location where you have downloaded the custom trained FlowerModel.tflite earlier.

Note that the tooling will configure the module's dependency on your behalf with ML Model binding and all dependencies automatically inserted into your Android module's build.gradle file.

4. Click Finish.
5. You will see the following at the end. The FlowerModel.tflite is successfully imported and it shows the high level information regarding the model including the input / output as well as some sample code to get you started.

6. Optional: Checking out all todo list

TODO list makes it easy to navigate to the exact location where you need to update the codelab. You can also use it in your Android project to remind yourself of future work. You can add todo items using code comments and type the keyword TODO. To access the list of TODOs, you can:

1. A great way to see what we are going to do is to check out the TODO list. To do that, select from the top menu bar View > Tool Windows > TODO

2. By default, it lists all TODOs in all modules which makes it a little confusing. We can sort out only the start TODOs by clicking on the group by button on the side of the TODO panel and choose Modules

3. Expand all the items under the start modules:

7. Run the custom model with TensorFlow Lite

1. Click on TODO 1 in the TODO list or open the MainActivity.kt file and locate TODO 1, initialize the model by adding this line:

private class ImageAnalyzer(ctx: Context, private val listener: RecognitionListener) :
        ImageAnalysis.Analyzer {

  ...
  // TODO 1: Add class variable TensorFlow Lite Model
  private val flowerModel = FlowerModel.newInstance(ctx)

  ...
}

2. Inside the analyze method for the CameraX Analyzer, we need to convert the camera input ImageProxy into a Bitmap and create a TensorImage object for the inference process.

Note that current tooling requires image input to be in Bitmap format.

• This means if the input is a file rather than ImageProxy, you can feed it directly to the method fromBitmap as a Bitmap object.
• If you are interested in how ImageProxy is converted into Bitmap, please check out the method toBitmap and the YuvToRgbConverter. We expect the needs for these methods to be temporary as the team works towards ImageProxy support for TensorImage.

override fun analyze(imageProxy: ImageProxy) {
  ...
  // TODO 2: Convert Image to Bitmap then to TensorImage
  val tfImage = TensorImage.fromBitmap(toBitmap(imageProxy))
  ...
}

3. Process the image and perform the following operations on the result:

• Descendingly sort the results by probability under the attribute score with the highest probability first.
• Take the top k results as defined by the constant MAX_RESULT_DISPLAY. You can optionally vary the value of this variable to get more or less results.

override fun analyze(imageProxy: ImageProxy) {
  ...
  // TODO 3: Process the image using the trained model, sort and pick out the top results
  val outputs = flowerModel.process(tfImage)
      .probabilityAsCategoryList.apply {
          sortByDescending { it.score } // Sort with highest confidence first
      }.take(MAX_RESULT_DISPLAY) // take the top results

  ...
}

4. Convert the sorted and filtered results into data objects Recognition ready to be consumed by RecyclerView via Data Binding:

override fun analyze(imageProxy: ImageProxy) {
  ...
  // TODO 4: Converting the top probability items into a list of recognitions
  for (output in outputs) {
      items.add(Recognition(output.label, output.score))
  }
  ...
}

5. Comment out or delete the following lines which help generate the fake results we see before:

// START - Placeholder code at the start of the codelab. Comment this block of code out.
for (i in 0..MAX_RESULT_DISPLAY-1){
    items.add(Recognition("Fake label $i", Random.nextFloat()))
}
// END - Placeholder code at the start of the codelab. Comment this block of code out.

8. Run the app on a real device by selecting TFL_Classify.start and press the run button on the toolbar:

9. You will see the following screen on your phone with random numbers taking the place of where real results will be displayed:

8. Optional: Accelerate inference with GPU delegate

TensorFlow Lite supports several hardware accelerators to speed up inference on your mobile device. GPU is one of the accelerators that TensorFlow Lite can leverage through a delegate mechanism and it is fairly easy to use.

1. Open build.gradle under the start module or you can click on TODO 5 under the TODO list and add the following dependency:

// TODO 5: Optional GPU Delegates    
implementation 'org.tensorflow:tensorflow-lite-gpu:2.3.0'

Note: we are adding this import manually in this codelab but when you do this in your project, you can just tick on the second box in the import screen to add GPU acceleration dependency:

2. Go back to the MainActivity.kt file or click on TODO 6 in the TODO list. Replace the simple initiation of the flowerModel with the following: Get an instance of the GPU compatibility list and initialize the GPU depending on if it is one of the listed compatible GPUs. Otherwise, initiate 4 CPU threads to run the model instead:

private class ImageAnalyzer(ctx: Context, private val listener: RecognitionListener) :
        ImageAnalysis.Analyzer {
  ...

  // TODO 1: Add class variable TensorFlow Lite Model
  // Initializing the flowerModel by lazy so that it runs in the same thread when the process
  // method is called.
  private val flowerModel: FlowerModel by lazy{

    // TODO 6. Optional GPU acceleration
    val compatList = CompatibilityList()

    val options = if(compatList.isDelegateSupportedOnThisDevice){
        Log.d(TAG, "This device is GPU Compatible ")
        Model.Options.Builder().setDevice(Model.Device.GPU).build()
    } else {
        Log.d(TAG, "This device is GPU Incompatible ")
        Model.Options.Builder().setNumThreads(4).build()
    }

  ...
}

Note: there are multiple Model objects, choose the object org.tensorflow.lite.support.model.Model in your import

3. Change the model initializer to use this by adding options to the method input:

private class ImageAnalyzer(ctx: Context, private val listener: RecognitionListener) :
        ImageAnalysis.Analyzer {

  private val flowerModel: FlowerModel by lazy{

    ...

    // Initialize the Flower Model
    FlowerModel.newInstance(ctx, options)
  }
}

10. Run the app on a real device by selecting TFL_Classify.start and press the run button on the toolbar:

On a medium/high end mobile device, GPU is much faster than CPU. Low end devices tend to have slower GPUs, so the speedup you see will vary.

9. What Next?

Here are some links for more information:

• Try out other TFLite models compatible with ML Model binding from tfhub.dev.
• Learn more about TFLite from the docs on tensorflow.org and the code repo.
• Try some other TFLite pretrained models including a speech hotword detector and an on-device version of smart-reply.
• Learn more about TensorFlow in general with our getting started docs.