ML Kit is a mobile SDK that brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can easily implement the functionality you need in just a few lines of code. There's no need to have deep knowledge of neural networks or model optimization to get started.

How does it work?

ML Kit makes it easy to apply ML techniques in your apps by bringing Google's ML technologies, such as the Google Cloud Vision API, Mobile Vision, and TensorFlow Lite, together in a single SDK. Whether you need the power of cloud-based processing, the real-time capabilities of Mobile Vision's on-device models, or the flexibility of custom TensorFlow Lite models, ML Kit makes it possible with just a few lines of code.

This codelab will walk you through simple steps to add Object Detection and Tracking(ODT) for a given image into your existing iOS app. Please note that this codelab takes some shortcuts to highlight the MLKit ODT usage.

What you will build

In this codelab, you're going to build an iOS app with ML Kit for Firebase. Your app will use the ML Kit Object Detection API to detect and track objects in a given image.

In the end, you should see something similar to the image on the right.

What you'll learn

What you'll need

This codelab is focused on ML Kit. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.

Download the Code

Click the following link to download all the code for this codelab:

Download source code

Unpack the downloaded zip file. This will unpack a root folder (mlkit-ios) with all of the resources you will need. For this codelab, you will only need the resources in the object-detection subdirectory.

The object-detection subdirectory in the mlkit-ios repository contains two directories:

  1. Go to the Firebase console.
  2. Select Create New Project, and name your project "ML Kit Codelab."

Connect your iOS app

  1. From the overview screen of your new project,
    click Add Firebase to your iOS app.
  2. Enter the codelab's package name: com.google.firebase.codelab.object-detection

Add GoogleService-Info.plist file to your app

After adding the package name and selecting Continue, your browser automatically downloads a configuration file that contains all the necessary Firebase metadata for your app. Copy the GoogleService-Info.plist file into your project.

Add the dependencies for ML Kit with CocoaPods

CocoaPods is used to add the ML Kit dependencies to your app. Installation instructions for CocoaPods are available here. Once installed follow setup a Podfile in the command line.

Confirm the Podfile in your project adheres to below.

Podfile

platform :ios, '9.0'
use_frameworks!

pod 'Firebase/Analytics'
pod 'Firebase/MLVisionObjectDetection'

target 'MLKit-codelab' do
end

Install ML Kit Cocoa Pods

To be sure that all dependencies are available to your app, you should use the command line to install the ML Kit Cocoa Pods.

Command Line

# Make sure you in the root of your app
pod install
open MLKit-codelab.xcworkspace

Now that you have imported the project into XCode and configured the GoogleService-Info.plist, and added the dependencies for ML Kit, you are ready to run the app for the first time. Start the XCode Simulator, and click Run in Xcode.

The app should launch on your simulator. At this point, you should see a basic layout that has a picker which allows you to select between 6 images. In the next section, you add object detection to your app.

In this step, we will add functionality to your app to detect and track objects in images.

Import the MLVision module

Confirm the following import to your ViewController class exists.

ViewController.swift

import FirebaseMLVisionObjectDetection

Create an ObjectDetector

There are 3 options you can configure on the object detector:

This codelab is for single image object detect & classification, let's do that:

Add the following lazy properties to your ViewController class.

ViewController.swift

  private lazy var vision = Vision.vision()
  private lazy var options: VisionObjectDetectorOptions = {
    let options = VisionObjectDetectorOptions()
    options.shouldEnableClassification = true
    options.shouldEnableMultipleObjects = true
    options.detectorMode = .singleImage
    return options
  }()

  private lazy var objectDetector = vision.objectDetector(options: options)

Run on-device object detection on an image

Add the following to the runObjectDetection method of ViewController class:

ViewController.swift

 func runObjectDetection(with image: UIImage) {
    let visionImage = VisionImage(image: image)
    objectDetector.process(visionImage) { objects, error in
      self.processResult(from: objects, error: error)
    }
  }

The code above configures the text recognition detector and calls the function processResult(from:, error:) with the response.

Process the object detection response

Add the following code to processResult in the ViewController class to parse the results and display them in your app.

ViewController.swift

 func processResult(from objects: [VisionObject]?, error: Error?) {
    removeDetectionAnnotations()
    guard error == nil else {
      let errorString = error?.localizedDescription ?? Constants.detectionNoResultsMessage
      print("Object detection failed with error: \(errorString)")
      return
    }

    guard let objects = objects, !objects.isEmpty else {
      print("On-Device object detector returned no results.")
      return
    }

    let transform = self.transformMatrix()

    objects.forEach { object in
      drawFrame(object.frame, in: .green, transform: transform)

      let transformedRect = object.frame.applying(transform)
      UIUtilities.addRectangle(
        transformedRect,
        to: self.annotationOverlayView,
        color: .green
      )

      let label = UILabel(frame: transformedRect)
      label.numberOfLines = 2
      label.text = "Category: \(categories[object.classificationCategory.rawValue])\nConfidence: \(object.confidence ?? 0)"
      label.adjustsFontSizeToFitWidth = true
      self.annotationOverlayView.addSubview(label)
    }
  }

Run the app on the simulator

Now click Run in XCode. Once the app loads, click Detect button on the navigation bar.

Your app should now look like image below, showing the object detection results and bounding boxes overlaid on top of the original image.

Congratulations, you have just added on-device object detection to your app using ML Kit for Firebase! Now you can try object detection on the images from camera as well by clicking icon and Detect afterwards.

You have used ML Kit for Firebase to add Object Detection capabilities to your app, as you proceed, you would also want to enhance the model, as you could see that the default model could only recognize 5 categories, outside this 5 categories are all classified as "Unknown". It is also noticed that when images are crowded with objects, MLKit ODT might miss some objects, as you can see from the above final classification. Most cases the detection and classification results are pretty accurate and correct.

What we've covered

Next Steps

Learn More