TensorFlow is a multipurpose machine learning framework. TensorFlow can be used anywhere from training huge models across clusters in the cloud, to running models locally on an embedded system like your phone.

What you'll Learn

What you will build

A simple camera app that runs a TensorFlow image recognition program to identify flowers.

Image by Orazio Puccio

This tutorial recommends that you complete TensorFlow for Poets first, and use the network you build in that tutorial as a starting point for this one.

Most of this codelab will be using the terminal, , open it now.

Clone the Git repository

The following command will clone the Git repository containing the files required for this codelab:

git clone https://github.com/googlecodelabs/tensorflow-for-poets-2

Now cd into the directory of the clone you just created. That's where you will be working for the rest of this codelab:

cd tensorflow-for-poets-2
ls 

The repo contains two directories: android/, and scripts/

The android/ directory contains nearly all the files necessary to build the a simple Android app that classifies images as it reads them from the camera. The only files missing for the app are those defining the image classification model, which you will create in this tutorial.

The scripts/ directory contains the python scripts you'll be using throughout the tutorial. These include scripts to prepare, test and evaluate the model.

Copy your tf_files from TensorFlow for Poets

Now copy the tf_files directory from the first part, into your working directory:

cp -r ~/tf_files .
ls

Check the contents of the tf_files/ directory to make sure you have the required files:

ls -lah tf_files/

As a minimum starting point it should contain retrained_graph.pb, retrained_labels.txt, and a flower_photos/ directory to test the classifier on.

Or don't, I'm a codelab, not a cop

If for any reason the tf_files/ directory, from the first codelab, is unavailable the minimal set of files required for this codelab is also available in the git clone, on the tf_files branch:

git checkout tf_files
ls -lah tf_files/

Optional: Manual install

Setup Docker

Install Docker

If you don't have docker installed already you can download the installer here.

Test your Docker installation

To test your Docker installation try running the following command in the terminal :

docker run hello-world

This should output some text starting with:

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

Run and test the TensorFlow image

Now that you've confirmed that Docker is working, test out the TensorFlow image:

docker run -it tensorflow/tensorflow:1.1.0 bash

After downloading your prompt should change to root@xxxxxxx:/notebooks#.

Next check to confirm that your TensorFlow installation works by invoking Python from the container's command line:

# Your prompt should be "root@xxxxxxx:/notebooks" 
python

Once you have a python prompt, >>>, run the following code:

# python

import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session() # It will print some warnings here.
print(sess.run(hello))

This should print Hello TensorFlow! (and a couple of warnings after the tf.Session line).

Exit Docker

Now press Ctrl-d, on a blank line, once to exit python, and a second time to exit the docker image.

Launch Docker with the working directory visible

Now relaunch Docker with that directory shared as your working directory, and port number 6006 published for TensorBoard:

docker run -it \
  --publish 6006:6006 \
  --volume ${HOME}/tensorflow-for-poets-2:/tensorflow-for-poets-2 \
  --workdir /tensorflow-for-poets-2 \
  tensorflow/tensorflow:1.1.0 bash

Your prompt will change to root@xxxxxxxxx:/tensorflow-for-poets-2#

Next, verify that the model is producing sane results before starting to modifying it.

The scripts/ directory contains a simple command line script, label_image.py, to test the network.

The script takes 2 arguments. The first is the path to the image file to evaluate, the second is the path to the TensorFlow graph.pb file.

Now test the program on this picture of a daisy:

flower_photos/daisy/21652746_cc379e0eea_m.jpg

Image by Retinafunk


This command will run the label_image script on the above photograph, using your retrained tensorflow graph.

python -m scripts.label_image \
  tf_files/flower_photos/daisy/21652746_cc379e0eea_m.jpg \
  tf_files/retrained_graph.pb

The script will print the elapsed time, and the probability the model has assigned to each flower type. Something like this:

Elapsed time: 0.801284

daisy (score = 0.98922)
dandelion (score = 0.00591)
sunflowers (score = 0.00428)
tulips (score = 0.00043)
roses (score = 0.00017)

This should hopefully produce a sensible top label for your example. You'll be using this command to make sure you're still getting sensible results as you do further processing on the model file to prepare it for use in a mobile app.

Mobile devices have significant limitations. So any pre-processing that can be done to reduce an app's footprint is worth considering.

Limited libraries

One way the TensorFlow library is kept small, for mobile, is by only supporting the subset of operations that are commonly used during inference. This is a reasonable approach, as training is rarely conducted on mobile platforms. Similarly it also excludes support for Operations with large external dependencies.

You can see the list of supported ops in the tensorflow/contrib/makefile/tf_op_files.txt file.

One of the operations that isn't supported is DecodeJpeg. The current implementation relies on libjpeg which is painful to support on mobile and would increase the binary footprint. While, technically, TensorFlow could add new implementations that use the mobile OS's native image libraries, for most mobile applications you don't need to decode JPEGs because you're dealing directly with camera image buffers.

Unfortunately the Inception model that the retraining is based on includes a DecodeJpeg operation, shown in the TensorBoard screenshot below. It is simple to bypass an operation in a TensorFlow graph, for example you could feed your image to the Cast node just above it and the DecodeJpeg would never run. However, if your platform does not support an operation you will not even be able to load the graph without an error.

Optimize for inference

To avoid this problem, the TensorFlow installation includes a tool, optimize_for_inference, that removes all nodes that aren't needed for a given set of input and output nodes.

The script also does a few other optimizations that help speed up the model, such as merging explicit batch normalization operations into the convolutional weights to reduce the number of calculations. Here's how you run it:

python -m tensorflow.python.tools.optimize_for_inference \
  --input=tf_files/retrained_graph.pb \
  --output=tf_files/optimized_graph.pb \
  --input_names="Cast" \
  --output_names="final_result"

This, after about 30 seconds, creates a new file at tf_files/optimized_graph.pb.

Verify the optimized model

To check that it hasn't altered the output of the network, compare the label_image output for retrained_graph.pb and optimized_graph.pb:

python -m scripts.label_image \
  tf_files/flower_photos/daisy/21652746_cc379e0eea_m.jpg \
  tf_files/retrained_graph.pb
python -m scripts.label_image \
  tf_files/flower_photos/daisy/21652746_cc379e0eea_m.jpg \
  tf_files/optimized_graph.pb

When I run these commands I see no change in the output probabilities to 5 decimal places, and a 30% speedup.

Now run it yourself to confirm that you see similar results.

Investigate the changes with TensorBoard

There is a simple script included to allow you to count nodes in a graph, or nodes of a particular type to see how the graph was modified:

python -m scripts.count_ops tf_files/retrained_graph.pb 
python -m scripts.count_ops tf_files/retrained_graph.pb DecodeJpeg

python -m scripts.count_ops tf_files/optimized_graph.pb 
python -m scripts.count_ops tf_files/optimized_graph.pb DecodeJpeg

TensorBoard, however, provides a much more exploration friendly environment.

If you followed along for the first tutorial, you should have a tf_files/training_summaries/ directory (otherwise, just create the directory: mkdir tf_files/training_summaries/).

Now launch TensorBoard, in the background, and point it at that directory:

tensorboard --logdir tf_files/training_summaries &

TensorBoard, running in the background, may occasionally print the following warning to your terminal:

WARNING:tensorflow:path ../external/data/plugin/text/runs not found, sending 404.

These can be safely ignored.

Now add your two graphs as TensorBoard logs:

python -m scripts.graph_pb2tb tf_files/training_summaries/retrained \
  tf_files/retrained_graph.pb 

python -m scripts.graph_pb2tb tf_files/training_summaries/optimized \
  tf_files/optimized_graph.pb 

Now open TensorBoard, and navigate to the "Graph" tab. Then from the pick-list labeled "Run", on the left side, select "Retrained".

Explore the graph a little, then select "Optimized" from the "Run" menu.

From here you can confirm that the DecodeJpeg node is in fact removed. If you continue exploring you will find other differences, especially if you expand the various blocks.

Check the compression baseline

The retrained model is still 84MB in size at this point. That large download size may be a limiting factor for any app that includes it.

Every mobile app distribution system compresses the package before distribution. So test how much the graph can be compressed using the gzip command:

gzip -c tf_files/optimized_graph.pb > tf_files/optimized_graph.pb.gz

gzip -l tf_files/optimized_graph.pb.gz

Not much!

On its own, compression is not a huge help. For me this only shaves 7% off the model size. If you're familiar with how neural networks and compression work this should be unsurprising.

The majority of the space taken up by the graph is by the weights, which are large blocks of floating point numbers. Each weight has a slightly different floating point value, with very little regularity.

But compression works by exploiting regularity in the data, which explains the failure here.

Example: Quantize an Image

Images can also be thought of as large blocks of numbers. One simple technique for compressing images it to reduce the number of colors. You will do the same thing to your network weights, after I demonstrate the effect on an image.

Below I've used ImageMagick's convert utility to reduce an image to 32 colors. This reduces the image size by more than a factor of 5 (png has built in compression). I do not recommend using this method to compress your images.

24 bit color: 290KB

32 colors: 55KB

Image by Fabrizio Sciami

Quantize the network weights

Applying an almost identical process to your neural network weights has a similar effect. It gives a lot more repetition for the compression algorithm to take advantage of, while reducing the precision by a small amount (typically less than a 1% drop in precision).

It does this without any changes to the structure of the network, it simply quantizes the constants in place.

Now use the quantize_graph script to apply these changes:

(This script is from the TensorFlow repository, but it is not included in the default installation)

python -m scripts.quantize_graph \
  --input=tf_files/optimized_graph.pb \
  --output=tf_files/rounded_graph.pb \
  --output_node_names=final_result \
  --mode=weights_rounded

Now try compressing this quantized model:

gzip -c tf_files/rounded_graph.pb > tf_files/rounded_graph.pb.gz

gzip -l tf_files/rounded_graph.pb.gz

You should see a significant improvement. I get 73% compression instead of 7%!

Now before you continue verify that this hasn't had too negative an effect on the model's performance.

First manually compare the two models on an example image.

python -m scripts.label_image \
  tf_files/flower_photos/daisy/21652746_cc379e0eea_m.jpg \
  tf_files/optimized_graph.pb
python -m scripts.label_image \
  tf_files/flower_photos/daisy/21652746_cc379e0eea_m.jpg \
  tf_files/rounded_graph.pb

For me, on this input image, the output probabilities have each changed by less than one tenth of a percent (absolute).

Next verify the change on a larger slice if the data to see how it affects overall performance.

First evaluate the performance of the baseline model on the validation set. The last two lines of the output show the average performance. It may take a minute or two to get the results back.

python -m scripts.evaluate  tf_files/optimized_graph.pb

For me, optimized_graph.pb scores 90.8% accuracy, and 0.258 for cross entropy error (lower is better).

Now compare that with the performance of the model in rounded_graph.pb:

python -m scripts.evaluate  tf_files/rounded_graph.pb

You should see less than a 1% change in the model accuracy. With the reduced dataset you may see no change in accuracy at all.

These differences are far from statistically significant. The goal is simply to confirm that the model was clearly not broken by this change.

Add your model to the project

The demo project is configured to search for the rounded_graph.pb, and retrained_labels.txt files in the android/assets directory.

Copy the files into the expected location:

cp tf_files/rounded_graph.pb tf_files/retrained_labels.txt android/assets/ 

Install AndroidStudio

If you don't have it installed already, go install AndroidStudio.

Open the project with AndroidStudio

Open AndroidStudio. After it loads select " Open an existing Android Studio project" from this popup:

In the file selector choose, from your working directory: tensorflow-for-poets-2/android.

You will get a "Gradle Sync" popup, the first time you open the project, asking about using gradle wrapper. Click "OK".

Setup the android device

You can't load the app from android studio onto your phone unless you activate "developer mode" and "USB Debugging". This is a one time setup process.

Follow the instructions at how to geek.

Build and install the app

Hit play, , in Android Studio to start the build and install process.

Next you will need to select your phone from this popup:

Now allow the phone give the Tensorflow Demo to access your camera, and files:

Now the app is installed, click the app icon, , to launch it. You can hold the power and volume-down buttons to take a screenshot.

Now try a web search for flowers, point the camera at the computer screen, and see if those pictures are correctly classified.

It should look something like this:

Image by Orazio Puccio

Or have a friend take a picture of you and find out what kind of TensorFlower you are !