AI Magic Wand with TensorFlow Lite for Microcontrollers and Arduino

What you'll build

In this codelab, we'll learn to use TensorFlow Lite For Microcontrollers to run a deep learning model on the Arduino Nano 33 BLE. The microcontroller is turned into a digital "magic wand" by the user to wave and cast a variety of spells. As the user moves the wand, this complex, multidimensional sensor data that would be inscrutable to a human, is passed as an input to the model, which outputs a simple classification that alerts us if one of several movements has occurred.

606293a8df189362.gif

Machine Learning on Microcontrollers

Machine learning can be used to create intelligent tools that make users' lives easier, like Google Assistant. But often, these experiences require a lot of computation or resources that can include a powerful cloud server or a desktop. However, it's now possible to run machine learning inference on tiny, low-powered hardware, like microcontrollers.

Microcontrollers are extremely common, cheap, require very little energy, and are very reliable. They are part of all sorts of household devices: think appliances, cars, and toys. In fact, there are around 30 billion microcontroller-powered devices produced each year.

1360b61fbfa33657.jpeg

By bringing machine learning to tiny microcontrollers, we can boost the intelligence of billions of devices that we use in our lives, without relying on expensive hardware or reliable internet connections. Imagine smart appliances that can adapt to your daily routine, intelligent industrial sensors that understand the difference between problems and normal operation, and magical toys that can help kids learn in fun and delightful ways.

TensorFlow Lite For Microcontrollers (Software)

358ffdb9eb758b90.png

TensorFlow is Google's open source machine learning framework for training and running models. TensorFlow Lite is a software framework, an optimized version of TensorFlow, targeted to run tensorflow models on small, relatively low-powered devices such as mobile phones.

TensorFlow Lite For Microcontrollers is a software framework, an optimized version of TensorFlow, targeted to run tensorflow models on tiny, low-powered hardware such as microcontrollers. It adheres to constraints required in these embedded environments, i.e, it has a small binary size, it doesn't require operating system support, any standard C or C++ libraries, or dynamic memory allocation, etc.

Arduino Nano 33 BLE (Hardware)

9528e6f546af12d3.jpeg

Arduino is a popular open-source platform used for building electronic projects. It consists of:

  1. A physical programmable circuit board (often a microcontroller) such as the Arduino Nano 33 BLE used in this codelab.
  2. The Arduino IDE (Integrated Development Environment), software which is used to write and upload computer code to the physical board.

Arduino Nano 33 BLE is a microcontroller-based platform: a tiny computer on a single circuit board. It has a processor, memory, and I/O hardware that allows it to send and receive digital signals to other devices. Unlike a computer, a microcontroller is not powerful and doesn't usually run an operating system. Instead, they have small processors, not much memory and the programs you write run directly on the hardware. But because they are designed to be as simple as possible, a microcontroller can use very little energy.

What you'll do

  • Setup the Arduino Nano 33 BLE microcontroller and turn it into a digital "magic wand"
  • Setup the Arduino IDE and install the required libraries
  • Deploy the program to the device
  • Cast a variety of spells by waving the magic wand and view the predictions

What you'll need

Unpack your Arduino

Remove it from the box and pull it out of the packing foam. It's conductive, and can cause problems otherwise!

3ca1a01c24ded4ee.jpeg

Plug it into your laptop

  • Plug the MicroUSB cable into the socket in the chip.
  • Plug the other end of the cable into a USB socket on your laptop.
  • The bottom left LED on the Arduino (top left in the image below) should light up.

2af957cf3b9818bc.jpeg

Familiarize yourself with the workflow

d371bb4465711ff6.png

As shown in the diagram above, the TFLite interpreter on the Arduino periodically runs inference on the model. The model uses the processed accelerometer data as the input and outputs a prediction which suggests the gesture that has most likely occurred. Further, a desired output would be printed and the right LEDs would light up.

1. Download the Arduino IDE

In order to deploy the program to the Arduino Microcontroller, we use the Arduino IDE.

Download the Arduino IDE

After downloading it, install and open the Arduino IDE by clicking on the application whose icon looks like this: 23e96e5962c17f6.png

The initial landing page would open as follows:

762dd25921f170d8.png

2. Setup the Board Manager

  1. From the Arduino menu, select Tools -> Board: "Arduino .." —> Boards Manager
  2. Search for "Arduino Nano 33 BLE" and install the Arduino nRF528x Boards (Mbed OS). This will ensure that our Arduino Nano 33 BLE Microcontroller is supported by the Arduino IDE. 817c63346152eda9.png
  1. From the Arduino menu, select Tools -> Board: "Arduino .." -> "Arduino Nano 33 BLE"

ae4434ef3d332c5b.png

  1. Finally, verify that your selected board is "Arduino Nano 33 BLE" at the bottom right of the IDE.

df53c7c91598d2d9.png

3. Setup the Port

From the Arduino menu, select Tools -> Port: "/.../" -> /dev/... (Arduino Nano 33 BLE). You should see something similar to this:

75072c07b51846f9.png

4. Check the Board Connection

From the Arduino menu, select Tools -> Get Board Info. You should see something similar to this:

605d93fcc757a7e8.png

1. TensorFlow Arduino Library

This library contains all the TensorFlow Lite for Microcontroller examples, which includes the magic wand source code required for this codelab.

Download the TensorFlow Arduino Library

  1. From the Arduino menu, select Sketch -> Include Library -> Add .ZIP Library...
  2. Add the TensorFlow Arduino Library .zip you downloaded.

2. LSM9DS1 Arduino Library

This library allows you to read the accelerometer, magnetometer and gyroscope values from the LSM9DS1 IMU on your Arduino Nano 33 BLE Sense.

  1. From the Arduino menu, select Sketch -> Include -> Manage Libraries...
  2. Search and Install "Arduino_LSM9DS1". 2e959411b51ea8cd.png

1. Load the example

From the Arduino menu, select File -> Examples -> Arduino_TensorFlowLite -> magic_wand to load the sample code.

798dd83a106a895f.png

This will load the magic wand source code.

1037a3b3e3f1bf03.png

2. Build the example

Click the Upload button in the sketch window.

7f7a2933151415bf.png

After a few minutes you should see red text indicating flashing has completed. During upload, the right LED should pulse, and then go dark at the end.

3df1d0858c6e40a4.png

The magic wand can currently detect 3 gestures as shown below: a7ff78a28c5fe73.png

  1. Wing: Start from the upper left corner and carefully trace the letter "W" for two seconds.
  2. Ring: Start upright, move the wand in a clockwise circle for one second.
  3. Slope: Start by holding the wand facing upward, with the LEDs toward you. Move the wand down at an incline to the left and then horizontally to the right for one second.

The following image demonstrates two gestures. First, a Slope and then a Wing (use this as a reference for your demo).

606293a8df189362.gif

To run the demo, follow the instructions given below:

  1. With the USB plugged in, select Tools -> Serial Monitor from the Arduino menu. It will initially open a blank screen with no output.

528d3f8b9c9a2167.png

  1. Move the Arduino microcontroller to trace each of the above shapes carefully and see if the serial monitor detects the gesture.

9b7027b480a62c4a.png

  1. From the output in the serial monitor, we notice that the magic wand indeed detected all the shapes! You'll also notice that the right LED lights up.

Congratulations, you've successfully built your first gesture-recognizing "magic wand" on an Arduino microcontroller!

We hope you've enjoyed this brief introduction to development with TensorFlow Lite for Microcontrollers. The idea of deep learning on microcontrollers is new and exciting, and we encourage you to go out and experiment!

Reference docs

26699b18f2b199f.png

Thanks, and have fun building!