1. Introduction
ML Kit is a mobile SDK that brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can easily implement the functionality you need in just a few lines of code. There's no need to have deep knowledge of neural networks or model optimization to get started.
If you run into any issues (code bugs, grammatical errors, unclear wording, etc.) as you work through this codelab, please report the issue via the Report a mistake link in the lower left corner of the codelab.
How does it work?
ML Kit makes it easy to apply ML techniques in your apps by bringing Google's ML technologies, such as the Mobile Vision, and TensorFlow Lite, together in a single SDK. Whether you need the real-time capabilities of Mobile Vision's on-device models, or the flexibility of custom TensorFlow Lite models, ML Kit has you covered.
This codelab will walk you through creating your own Android app that can automatically detect text and facial features in an image.
What you will build
In this codelab, you're going to build an Android app with ML Kit. Your app will:
|
|
What you'll learn
- How to use the ML Kit SDK to easily add advanced Machine Learning capabilities such as text recognition and facial feature detection
What you'll need
- A recent version of Android Studio (v3.0+)
- Android Studio Emulator or a physical Android device
- The sample code
- Basic knowledge of Android development in Java
- Basic understanding of machine learning models
This codelab is focused on ML Kit. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.
2. Getting set up
Download the Code
Click the following link to download all the code for this codelab:
Unpack the downloaded zip file. This will unpack a root folder (mlkit-android-master) with all of the resources you will need. For this codelab, you will only need the resources in the vision subdirectory.
The vision subdirectory in the mlkit-android-master repository contains two directories:
starter—Starting code that you build upon in this codelab.- final—Completed code for the finished sample app.
3. Check the dependencies for ML Kit
Verify the dependencies for ML Kit
The following lines should already be added to the end of the build.gradle file in the app directory of your project (check to confirm):
build.gradle
dependencies {
// Face features
implementation 'com.google.mlkit:face-detection:16.0.0'
// Text features
implementation 'com.google.android.gms:play-services-mlkit-text-recognition:16.0.0'
}
These are the specific ML Kit dependencies that you need to implement the features in this codelab.
4. Run the starter app
Now that you have imported the project into Android Studio and checked the dependencies for ML Kit, you are ready to run the app for the first time. Start the Android Studio emulator, and click Run ( 
) in the Android Studio toolbar.
The app should launch on your emulator. At this point, you should see a basic layout that has a drop down field which allows you to select between 3 images. In the next section, you will add text recognition to your app to identify text in images.
5. Add on-device text recognition
In this step, we will add the functionality to your app to recognize text in images.
Set up and run on-device text recognition on an image
Add the following to the runTextRecognition method of MainActivity class:
MainActivity.java
private void runTextRecognition() {
InputImage image = InputImage.fromBitmap(mSelectedImage, 0);
TextRecognizer recognizer = TextRecognition.getClient();
mTextButton.setEnabled(false);
recognizer.process(image)
.addOnSuccessListener(
new OnSuccessListener<Text>() {
@Override
public void onSuccess(Text texts) {
mTextButton.setEnabled(true);
processTextRecognitionResult(texts);
}
})
.addOnFailureListener(
new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
// Task failed with an exception
mTextButton.setEnabled(true);
e.printStackTrace();
}
});
}
The code above configures the text recognition detector and calls the function processTextRecognitionResult with the response.
Process the text recognition response
Add the following code to processTextRecognitionResult in the MainActivity class to parse the results and display them in your app.
MainActivity.java
private void processTextRecognitionResult(Text texts) {
List<Text.TextBlock> blocks = texts.getTextBlocks();
if (blocks.size() == 0) {
showToast("No text found");
return;
}
mGraphicOverlay.clear();
for (int i = 0; i < blocks.size(); i++) {
List<Text.Line> lines = blocks.get(i).getLines();
for (int j = 0; j < lines.size(); j++) {
List<Text.Element> elements = lines.get(j).getElements();
for (int k = 0; k < elements.size(); k++) {
Graphic textGraphic = new TextGraphic(mGraphicOverlay, elements.get(k));
mGraphicOverlay.add(textGraphic);
}
}
}
}
Run the app on the emulator
Now click Run ( ) in the Android Studio toolbar. Once the app loads, make sure that Test Image 1(Text) is selected in the drop down field and click on the FIND TEXT button.
Your app should now look like the image below, showing the text recognition results and bounding boxes overlaid on top of the original image.
Photo: Kai Schreiber / Wikimedia Commons / CC BY-SA 2.0
Congratulations, you have just added on-device text recognition to your app using ML Kit! On-device text recognition is great for many use cases as it works even when your app doesn't have internet connectivity and is fast enough to use on still images as well as live video frames.
6. Add on-device face contour detection
In this step, we will add functionality to your app to recognize the contours of faces in images.
Set up and run on-device face contour detection on an image
Add the following to the runFaceContourDetection method of MainActivity class:
MainActivity.java
private void runFaceContourDetection() {
InputImage image = InputImage.fromBitmap(mSelectedImage, 0);
FaceDetectorOptions options =
new FaceDetectorOptions.Builder()
.setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
.setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
.build();
mFaceButton.setEnabled(false);
FaceDetector detector = FaceDetection.getClient(options);
detector.process(image)
.addOnSuccessListener(
new OnSuccessListener<List<Face>>() {
@Override
public void onSuccess(List<Face> faces) {
mFaceButton.setEnabled(true);
processFaceContourDetectionResult(faces);
}
})
.addOnFailureListener(
new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
// Task failed with an exception
mFaceButton.setEnabled(true);
e.printStackTrace();
}
});
}
The code above configures the face contour detector and calls the function processFaceContourDetectionResult with the response.
Process the face contour detection response
Add the following code to processFaceContourDetectionResult in the MainActivity class to parse the results and display them in your app.
MainActivity.java
private void processFaceContourDetectionResult(List<Face> faces) {
// Task completed successfully
if (faces.size() == 0) {
showToast("No face found");
return;
}
mGraphicOverlay.clear();
for (int i = 0; i < faces.size(); ++i) {
Face face = faces.get(i);
FaceContourGraphic faceGraphic = new FaceContourGraphic(mGraphicOverlay);
mGraphicOverlay.add(faceGraphic);
faceGraphic.updateFace(face);
}
}
Run the app on the emulator
Now click Run ( ) in the Android Studio toolbar. Once the app loads, make sure that Test Image 2 (Face) is selected in the drop down field and click on the FIND FACE CONTOUR button.
Your app should now look like the image below, showing the face contour detection results and showing the contours of the face as points overlaid on top of the original image.
Congratulations, you have just added on-device face contour detection to your app using ML Kit! On-device face contour detection is great for many use cases as it works even when your app doesn't have internet connectivity and is fast enough to use on still images as well as live video frames.
7. Congratulations!
You have successfully used ML Kit to easily add advanced machine learning capabilities to your app.
What we've covered
- How to add ML Kit to your Android app
- How to use on-device text recognition in ML Kit to find text in images
- How to use on-device face contour in ML Kit to identify face features in images
Next Steps
- Use ML Kit in your own Android app!