Recent progress in machine learning has made it relatively easy for computers to recognize objects in images. In this codelab, we will walk you through an end-to-end journey building an image classification model that can recognize different types of objects, then deploy the model on Android and iOS app. ML Kit and AutoML allow you to build and deploy the model at scale without any machine learning expertise.
ML Kit is a mobile SDK that brings Google's machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Whether you're new or experienced in machine learning, you can easily implement the functionality you need in just a few lines of code. There are several APIs that can recognize text, faces etc. that you can use out-of-the-box. However, if you need to recognize objects that are not supported by the APIs, such as recognizing different types of flowers from an image, then you need to train your own model. This is where AutoML can help you.
Cloud AutoML is a suite of machine learning products that enables developers with limited machine learning expertise to train high-quality models specific to their business needs, by leveraging Google's state-of-the-art transfer learning, and Neural Architecture Search technology.
In this codelab, we will use AutoML Vision Edge in ML Kit to train a flower classification model. The model is trained in the cloud, but is then bundled or downloaded by the app to run inferences fully on-device.
Download a zip archive that contains the source code for this codelab and a training dataset. Extract the archive in your local machine.
google-services.jsonconfig file, and put it in the Android app at
GoogleService-Info.plistconfig file and follow the instructions to put it in the iOS app at
To train a model to recognize different types of objects, you have to prepare a set of images and label each of them. We have created an archive of creative-commons licensed flower photos for you to use in this codelab.
The dataset is packaged a zip file called
flower_photos.zip that is included in the zip archive you downloaded in the previous step.
If you extract the
flower_photos.zip file, you will see that the dataset contains images of 5 flower types: dandelion, daisy, tulips, sunflowers and roses that are organized into folders named after the flowers. This is a handy way to create a training dataset to feed to AutoML and train an image classification model.
There are 200 images for each flower type in this training dataset. You only need a minimum of 10 images per class to train a model. However, more training images will generally lead to better models.
flower_photos.zipfile that you downloaded in the previous step to import the flower training dataset.
As our model will run on a mobile device with limited computing power and storage, we have to be mindful about not only the model's accuracy but also its size and speed. There is always a trade off between model accuracy, latency (i.e how long it takes to classify an image) and model size. Generally a model with higher accuracy is also larger and will take longer to classify an image.
AutoML provides you several options: you can choose to optimize for accuracy, or to optimize for latency and model size, or to balance between them. You also can choose how long you allow the model to train. Larger datasets need to train for longer.
Here are the steps if you want to train the model yourself.
You only need to add the model to the sample apps and they will work out of the box. For a full guide on how to integrate ML Kit AutoML to your app, please see our documentation (Android, iOS). Code that interacts with ML Kit SDK is in the
ImageClassifier.swift file, respectively, so you can start from there to explore how the apps work.
There are two options to deploy the model: local and remote.
pod installto download dependencies via Cocoapods
open MLVisionExample.xcworkspace/to open the project workspace in Xcode.
ML Kit's remote model allows you to not include Tensorflow Lite models in your app binary but download it on-demand from Firebase when needed. Remote models have several benefits over local models:
In this step, we will publish a remote model and use it in the sample apps. Please make sure that you have finished training your model in step 4 of this codelab.
The sample apps are configured to use remote model if it is available. After you have published the remote model, you only need to rerun the apps to trigger model download. You can verify that the app is using the remote model by looking at the "Source" value in the footer of the app screen. See the "Troubleshooting" section below if it does not work.
If the sample app is still using the local model, please verify that the remote model name is set correctly inside the code.
ImageClassifier.ktand find this block.
val remoteModel = FirebaseRemoteModel.Builder(REMOTE_MODEL_NAME).build()
ImageClassifier.swiftand find this block,
return RemoteModel( name: Constant.remoteAutoMLModelName, allowsModelUpdates: true, initialConditions: initialConditions, updateConditions: updateConditions )
You have gone through an end-to-end journey of training an image classification model with your own training data using AutoML, and then use the model in a mobile app using ML Kit.
Please see our documentation to learn how to integrate AutoML Vision Edge in ML Kit to your own app.
You also can try our ML Kit sample apps to see other features of Firebase ML Kit.