Recognize text and facial features with ML Kit: Android

1. Introduction

ML Kit is a mobile SDK that brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can easily implement the functionality you need in just a few lines of code. There's no need to have deep knowledge of neural networks or model optimization to get started.

If you run into any issues (code bugs, grammatical errors, unclear wording, etc.) as you work through this codelab, please report the issue via the Report a mistake link in the lower left corner of the codelab.

How does it work?

ML Kit makes it easy to apply ML techniques in your apps by bringing Google's ML technologies, such as the Mobile Vision, and TensorFlow Lite, together in a single SDK. Whether you need the real-time capabilities of Mobile Vision's on-device models, or the flexibility of custom TensorFlow Lite models, ML Kit has you covered.

This codelab will walk you through creating your own Android app that can automatically detect text and facial features in an image.

What you will build

In this codelab, you're going to build an Android app with ML Kit. Your app will:

  • Use the ML Kit Text Recognition API to detect text in images
  • Use the ML Kit Face Contour API to identify facial features in images

What you'll learn

  • How to use the ML Kit SDK to easily add advanced Machine Learning capabilities such as text recognition and facial feature detection

What you'll need

  • A recent version of Android Studio (v3.0+)
  • Android Studio Emulator or a physical Android device
  • The sample code
  • Basic knowledge of Android development in Java
  • Basic understanding of machine learning models

This codelab is focused on ML Kit. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.

2. Getting set up

Download the Code

Click the following link to download all the code for this codelab:

Unpack the downloaded zip file. This will unpack a root folder (mlkit-android-master) with all of the resources you will need. For this codelab, you will only need the resources in the vision subdirectory.

The vision subdirectory in the mlkit-android-master repository contains two directories:

  • android_studio_folder.pngstarter—Starting code that you build upon in this codelab.
  • android_studio_folder.pngfinal—Completed code for the finished sample app.

3. Check the dependencies for ML Kit

Verify the dependencies for ML Kit

The following lines should already be added to the end of the build.gradle file in the app directory of your project (check to confirm):

build.gradle

dependencies {
  // Face features
  implementation 'com.google.mlkit:face-detection:16.0.0'

  // Text features
  implementation 'com.google.android.gms:play-services-mlkit-text-recognition:16.0.0'
}

These are the specific ML Kit dependencies that you need to implement the features in this codelab.

4. Run the starter app

Now that you have imported the project into Android Studio and checked the dependencies for ML Kit, you are ready to run the app for the first time. Start the Android Studio emulator, and click Run ( execute.png) in the Android Studio toolbar.

The app should launch on your emulator. At this point, you should see a basic layout that has a drop down field which allows you to select between 3 images. In the next section, you will add text recognition to your app to identify text in images.

5. Add on-device text recognition

In this step, we will add the functionality to your app to recognize text in images.

Set up and run on-device text recognition on an image

Add the following to the runTextRecognition method of MainActivity class:

MainActivity.java

private void runTextRecognition() {
   InputImage image = InputImage.fromBitmap(mSelectedImage, 0);
   TextRecognizer recognizer = TextRecognition.getClient();
   mTextButton.setEnabled(false);
   recognizer.process(image)
           .addOnSuccessListener(
                   new OnSuccessListener<Text>() {
                       @Override
                       public void onSuccess(Text texts) {
                           mTextButton.setEnabled(true);
                           processTextRecognitionResult(texts);
                       }
                   })
           .addOnFailureListener(
                   new OnFailureListener() {
                       @Override
                       public void onFailure(@NonNull Exception e) {
                           // Task failed with an exception
                           mTextButton.setEnabled(true);
                           e.printStackTrace();
                       }
                   });
}

The code above configures the text recognition detector and calls the function processTextRecognitionResult with the response.

Process the text recognition response

Add the following code to processTextRecognitionResult in the MainActivity class to parse the results and display them in your app.

MainActivity.java

private void processTextRecognitionResult(Text texts) {
   List<Text.TextBlock> blocks = texts.getTextBlocks();
   if (blocks.size() == 0) {
       showToast("No text found");
       return;
   }
   mGraphicOverlay.clear();
   for (int i = 0; i < blocks.size(); i++) {
       List<Text.Line> lines = blocks.get(i).getLines();
       for (int j = 0; j < lines.size(); j++) {
           List<Text.Element> elements = lines.get(j).getElements();
           for (int k = 0; k < elements.size(); k++) {
               Graphic textGraphic = new TextGraphic(mGraphicOverlay, elements.get(k));
               mGraphicOverlay.add(textGraphic);

           }
       }
   }
}

Run the app on the emulator

Now click Run ( execute.png) in the Android Studio toolbar. Once the app loads, make sure that Test Image 1(Text) is selected in the drop down field and click on the FIND TEXT button.

Your app should now look like the image below, showing the text recognition results and bounding boxes overlaid on top of the original image.

10b12d0e8822eaf5.png

Photo: Kai Schreiber / Wikimedia Commons / CC BY-SA 2.0

Congratulations, you have just added on-device text recognition to your app using ML Kit! On-device text recognition is great for many use cases as it works even when your app doesn't have internet connectivity and is fast enough to use on still images as well as live video frames.

6. Add on-device face contour detection

In this step, we will add functionality to your app to recognize the contours of faces in images.

Set up and run on-device face contour detection on an image

Add the following to the runFaceContourDetection method of MainActivity class:

MainActivity.java

private void runFaceContourDetection() {
   InputImage image = InputImage.fromBitmap(mSelectedImage, 0);
   FaceDetectorOptions options =
           new FaceDetectorOptions.Builder()
                   .setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
                   .setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
                   .build();

   mFaceButton.setEnabled(false);
   FaceDetector detector = FaceDetection.getClient(options);
   detector.process(image)
           .addOnSuccessListener(
                   new OnSuccessListener<List<Face>>() {
                       @Override
                       public void onSuccess(List<Face> faces) {
                           mFaceButton.setEnabled(true);
                           processFaceContourDetectionResult(faces);
                       }
                   })
           .addOnFailureListener(
                   new OnFailureListener() {
                       @Override
                       public void onFailure(@NonNull Exception e) {
                           // Task failed with an exception
                           mFaceButton.setEnabled(true);
                           e.printStackTrace();
                       }
                   });

}

The code above configures the face contour detector and calls the function processFaceContourDetectionResult with the response.

Process the face contour detection response

Add the following code to processFaceContourDetectionResult in the MainActivity class to parse the results and display them in your app.

MainActivity.java

private void processFaceContourDetectionResult(List<Face> faces) {
   // Task completed successfully
   if (faces.size() == 0) {
       showToast("No face found");
       return;
   }
   mGraphicOverlay.clear();
   for (int i = 0; i < faces.size(); ++i) {
       Face face = faces.get(i);
       FaceContourGraphic faceGraphic = new FaceContourGraphic(mGraphicOverlay);
       mGraphicOverlay.add(faceGraphic);
       faceGraphic.updateFace(face);
   }
}

Run the app on the emulator

Now click Run ( execute.png) in the Android Studio toolbar. Once the app loads, make sure that Test Image 2 (Face) is selected in the drop down field and click on the FIND FACE CONTOUR button.

Your app should now look like the image below, showing the face contour detection results and showing the contours of the face as points overlaid on top of the original image.

f9ff2fcbf63f0f3b.png

Congratulations, you have just added on-device face contour detection to your app using ML Kit! On-device face contour detection is great for many use cases as it works even when your app doesn't have internet connectivity and is fast enough to use on still images as well as live video frames.

7. Congratulations!

You have successfully used ML Kit to easily add advanced machine learning capabilities to your app.

What we've covered

  • How to add ML Kit to your Android app
  • How to use on-device text recognition in ML Kit to find text in images
  • How to use on-device face contour in ML Kit to identify face features in images

Next Steps

  • Use ML Kit in your own Android app!

Learn More