ML Kit is a mobile SDK that brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can easily implement the functionality you need in just a few lines of code. There's no need to have deep knowledge of neural networks or model optimization to get started. On the other hand, if you are an experienced ML developer, see the Custom Machine Learning Models with ML Kit codelab to learn how ML Kit makes it easy to use your custom TensorFlow Lite models in your mobile apps.

How does it work?

ML Kit makes it easy to apply ML techniques in your apps by bringing Google's ML technologies, such as the Google Cloud Vision API, Mobile Vision, and TensorFlow Lite, together in a single SDK. Whether you need the power of cloud-based processing, the real-time capabilities of Mobile Vision's on-device models, or the flexibility of custom TensorFlow Lite models, ML Kit makes it possible with just a few lines of code.

This codelab will walk you through creating your own Android app that can automatically scan and read the contents of a provided barcode.

What you will build

In this codelab, you're going to build an Android app with Firebase ML Kit. Your app will:

  • Utilize the ML Kit Barcode Scanning API to scan and process barcodes without requiring an internet connection.

What you'll learn

What you'll need

This codelab is focused on ML Kit. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.

Download the Code

Click the following link to download all the code for this codelab:

Download source code

Unpack the downloaded zip file. This will unpack a root folder (mlkit-android) with all of the resources you will need. For this codelab, you will only need the resources in the barcode-scanning subdirectory.

The barcode-scanning subdirectory in the mlkit repository contains two directories:

  1. Go to the Firebase console.
  2. Select Create New Project, and name your project "ML Kit Codelab."

Connect your Android app

  1. From the overview screen of your new project,
    click Add Firebase to your Android app.
  2. Enter the codelab's package name: com.google.firebase.codelab.barcode-scanning.

Add google-services.json file to your app

After adding the package name and selecting Continue, your browser automatically downloads a configuration file that contains all the necessary Firebase metadata for your app. Copy the google-services.json file into the app directory in your project.

Add the dependencies for ML Kit and the google-services plugin to your app

The google-services plugin uses the google-services.json file to configure your application to use Firebase, and the ML Kit dependencies allow you to integrate the ML Kit SDK in your app. The following lines should already be added to the end of the build.gradle file in the app directory of your project (check to confirm):

build.gradle

dependencies {
  // ...
  implementation 'com.google.firebase:firebase-ml-vision:17.0.0'
}
apply plugin: 'com.google.gms.google-services'

Sync your project with gradle files

To be sure that all dependencies are available to your app, you should sync your project with gradle files at this point. Select Sync Project with Gradle Files () from the Android Studio toolbar.

Now that you have imported the project into Android Studio and configured the google-services plugin with your JSON file, and added the dependencies for ML Kit, you are ready to run the app for the first time. Start the Android Studio emulator, and click Run () in the Android Studio toolbar.

The app should launch on your emulator. At this point, you should see a basic layout that has a Camera preview along with a FloatingActionButton to capture the image that's currently being shown in the preview.

In this step, we will add functionality to your app to detect and scan barcodes from images.

Set up and run on-device Barcode Scanning on an image

When it comes to the implementation, the APIs provided by MLKit have 3 main steps :

Step 1 : Creating a FirebaseVisionImage object using the captured Bitmap

Step 2 (Optional) : Determine what kind of Barcodes you would like to scan by creating an object of the FirebaseVisionBarcodeDetectorOptions class.

Step 2 : Creating the detector for the API. For Barcode Scanner API, this will be FirebaseVisionBarcodeDetector.

Step 3 : Running the detector over the FirebaseVisionImage and attach onSuccess, onFailure and onComplete listeners.

Add the following code snippet to the runBarcodeScanner method of BarcodeScannerActivity class:

BarcodeScannerActivity.kt

   private fun runBarcodeScanner(bitmap: Bitmap) {
        //Create a FirebaseVisionImage
        val image = FirebaseVisionImage.fromBitmap(bitmap)

        //Optional : Define what kind of barcodes you want to scan
        val options = FirebaseVisionBarcodeDetectorOptions.Builder()
                .setBarcodeFormats(
                        //Detect all kind of barcodes
                        FirebaseVisionBarcode.FORMAT_ALL_FORMATS
                        //Or specify which kind of barcode you want to detect
                        /*
                            FirebaseVisionBarcode.FORMAT_QR_CODE,
                        FirebaseVisionBarcode.FORMAT_AZTEC
                         */
                )
                .build()

        //Get access to an instance of FirebaseBarcodeDetector
        val detector = FirebaseVision.getInstance().getVisionBarcodeDetector(options)

        //Use the detector to detect the labels inside the image
        detector.detectInImage(image)
                .addOnSuccessListener {
                    qrList.clear()
                    adapter.notifyDataSetChanged()
                    // Task completed successfully
                    for (firebaseBarcode in it) {
                        when (firebaseBarcode.valueType) {
                            //Handle the URL here
                            FirebaseVisionBarcode.TYPE_URL ->
                                qrList.add(QrCode("URL", firebaseBarcode.displayValue))
                            // Handle the contact info here, i.e. address, name, phone, etc.
                            FirebaseVisionBarcode.TYPE_CONTACT_INFO ->
                                qrList.add(QrCode("Contact", firebaseBarcode.contactInfo?.title))
                            // Handle the wifi here, i.e. firebaseBarcode.wifi.ssid, etc.
                            FirebaseVisionBarcode.TYPE_WIFI ->
                                qrList.add(QrCode("WiFi", firebaseBarcode.wifi?.ssid))
                            // Handle the driver license barcode here, i.e. City, Name, Expiry, etc.
                            FirebaseVisionBarcode.TYPE_DRIVER_LICENSE ->
                                qrList.add(QrCode("Driver License", firebaseBarcode.driverLicense?.licenseNumber))
                            //Handle more types
                            else ->
                                qrList.add(QrCode("Generic", firebaseBarcode.displayValue))
                            //None of the above type was detected, so extract the value from the barcode
                        }
                    }
                    adapter.notifyDataSetChanged()
                    progressBar.visibility = View.GONE
                    sheetBehavior.setState(BottomSheetBehavior.STATE_EXPANDED)
                }
                .addOnFailureListener {
                    // Task failed with an exception
                    progressBar.visibility = View.GONE
                    Toast.makeText(baseContext, "Sorry, something went wrong!", Toast.LENGTH_SHORT).show()
                }
                .addOnCompleteListener {
                    progressBar.visibility = View.GONE
                }
    }

The code above does the following things :

  1. Creates the FirebaseVisionImage
  2. Creates a FirebaseVisionBarcodeDetectorOptions object which is set to detect all types of barcodes.
  3. Configures the image label detector and then runs it over the image to get a list of FirebaseVisionBarcode objects containing the data associated with the Barcode.
  4. The code also loops through the list of detected barcodes and extracts the appropriate data from them depending on the type of Barcode detected.

Understanding the response

The detector's onSuccessListener method returns a List of FirebaseVisionBarcode object which in turn contains data associated with the detected Barcode.
The data might vary from Barcode to Barcode, but some of them are discussed below :

Bounding Box : This is the bounding rectangle which contains the barcode.

Display Value : This is the data inside the barcode in a human readable form.

Wifi : This contains the data associated with a wifi network (SSID, Password) for a barcode containing WiFi information of a network.

Driver License : This contains the data associated with a driver's license (Number, City, Name, Expiry, etc.) for a barcode containing information about the driver license.

URL : This contains the data associated with a URL (title, link) for a barcode containing information about a URL

This list of detected barcodes is then passed on to the adapter where it is populated and displayed inside a recyclerview.

Run the app on the emulator

Now click Run () in the Android Studio toolbar. Once the app loads, point your camera to an object and press the FloatingActionButton with the camera icon and wait for the app to process the image.

Once done, your app should now look like image below, showing a list of detected Barcodes alongwith their value.

Photo: Harshit Dwivedi

Congratulations, you have just added on-device object detection to your app using ML Kit for Firebase! Barcode Scanning API is great for many use cases as it works even when your app doesn't have internet connectivity.

You have used ML Kit for Firebase to easily add advanced machine learning capabilities to your app.

What we've covered

Next Steps

Learn More