AI Magic Wand with TensorFlow Lite for Microcontrollers and Arduino

1. Introduction

What you'll build

In this codelab, we'll learn to use TensorFlow Lite For Microcontrollers to run a deep learning model on the Arduino Nano 33 BLE. The microcontroller is turned into a digital "magic wand" by the user to wave and cast a variety of spells. As the user moves the wand, this complex, multidimensional sensor data that would be inscrutable to a human, is passed as an input to the model, which outputs a simple classification that alerts us if one of several movements has occurred.

Machine Learning on Microcontrollers

Machine learning can be used to create intelligent tools that make users' lives easier, like Google Assistant. But often, these experiences require a lot of computation or resources that can include a powerful cloud server or a desktop. However, it's now possible to run machine learning inference on tiny, low-powered hardware, like microcontrollers.

Microcontrollers are extremely common, cheap, require very little energy, and are very reliable. They are part of all sorts of household devices: think appliances, cars, and toys. In fact, there are around 30 billion microcontroller-powered devices produced each year.

By bringing machine learning to tiny microcontrollers, we can boost the intelligence of billions of devices that we use in our lives, without relying on expensive hardware or reliable internet connections. Imagine smart appliances that can adapt to your daily routine, intelligent industrial sensors that understand the difference between problems and normal operation, and magical toys that can help kids learn in fun and delightful ways.

TensorFlow Lite For Microcontrollers (Software)

TensorFlow is Google's open source machine learning framework for training and running models. TensorFlow Lite is a software framework, an optimized version of TensorFlow, targeted to run tensorflow models on small, relatively low-powered devices such as mobile phones.

TensorFlow Lite For Microcontrollers is a software framework, an optimized version of TensorFlow, targeted to run tensorflow models on tiny, low-powered hardware such as microcontrollers. It adheres to constraints required in these embedded environments, i.e, it has a small binary size, it doesn't require operating system support, any standard C or C++ libraries, or dynamic memory allocation, etc.

Arduino Nano 33 BLE (Hardware)

Arduino is a popular open-source platform used for building electronic projects. It consists of:

1. A physical programmable circuit board (often a microcontroller) such as the Arduino Nano 33 BLE used in this codelab.
2. The Arduino IDE (Integrated Development Environment), software which is used to write and upload computer code to the physical board.

Arduino Nano 33 BLE is a microcontroller-based platform: a tiny computer on a single circuit board. It has a processor, memory, and I/O hardware that allows it to send and receive digital signals to other devices. Unlike a computer, a microcontroller is not powerful and doesn't usually run an operating system. Instead, they have small processors, not much memory and the programs you write run directly on the hardware. But because they are designed to be as simple as possible, a microcontroller can use very little energy.

Note: We use the Arduino Nano 33 BLE microcontroller as it is compatible with TensorFlow Lite for Microcontrollers, has a three-axis accelerometer which can be used to capture and predict gestures, and is small and lightweight—which make it ideal for building a magic wand!

What you'll do

• Setup the Arduino Nano 33 BLE microcontroller and turn it into a digital "magic wand"
• Setup the Arduino IDE and install the required libraries
• Deploy the program to the device
• Cast a variety of spells by waving the magic wand and view the predictions

What you'll need

• Linux, MacOS or Windows laptop
• Arduino Nano BLE Sense 33 (without headers)
• Micro USB cable (If you're on a USB-C laptop, instead get a USB-C to Micro USB cable)
• (optional) Stick, about 12 inches (30cm long)
• (optional) Sticky tape

2. Setup the Arduino Microcontroller

Unpack your Arduino

Remove it from the box and pull it out of the packing foam. It's conductive, and can cause problems otherwise!

Plug it into your laptop

• Plug the MicroUSB cable into the socket in the chip.
• Plug the other end of the cable into a USB socket on your laptop.
• The bottom left LED on the Arduino (top left in the image below) should light up.

Familiarize yourself with the workflow

As shown in the diagram above, the TFLite interpreter on the Arduino periodically runs inference on the model. The model uses the processed accelerometer data as the input and outputs a prediction which suggests the gesture that has most likely occurred. Further, a desired output would be printed and the right LEDs would light up.

3. Setup the Arduino IDE

1. Download the Arduino IDE

In order to deploy the program to the Arduino Microcontroller, we use the Arduino IDE.

Caution: Currently, we do not support the Arduino Web Editor to build and deploy the program. You are required to install and run the Arduino IDE locally on your machine.

Download the Arduino IDE

After downloading it, install and open the Arduino IDE by clicking on the application whose icon looks like this:

The initial landing page would open as follows:

2. Setup the Board Manager

1. From the Arduino menu, select Tools -> Board: "Arduino .." —> Boards Manager
2. Search for "Arduino Nano 33 BLE" and install the Arduino nRF528x Boards (Mbed OS). This will ensure that our Arduino Nano 33 BLE Microcontroller is supported by the Arduino IDE.

Note: The Board Manager you see may not match the one shown in the image above and that's all right. Ensure that the one you install has "Arduino Nano 33 BLE" listed under the "Boards included in this package:" section.

3. From the Arduino menu, select Tools -> Board: "Arduino .." -> "Arduino Nano 33 BLE"

4. Finally, verify that your selected board is "Arduino Nano 33 BLE" at the bottom right of the IDE.

3. Setup the Port

From the Arduino menu, select Tools -> Port: "/.../" -> /dev/... (Arduino Nano 33 BLE). You should see something similar to this:

Note: Troubleshooting this is a black art across Linux, MacOS or Windows machines. A good place to start is here: https://www.arduino.cc/en/Guide/Troubleshooting#toc16. Also, ensure that the board is plugged into the computer/laptop.

4. Check the Board Connection

From the Arduino menu, select Tools -> Get Board Info. You should see something similar to this:

4. Install Libraries

1. TensorFlow Arduino Library

This library contains all the TensorFlow Lite for Microcontroller examples, which includes the magic wand source code required for this codelab.

Download the TensorFlow Arduino Library

1. From the Arduino menu, select Sketch -> Include Library -> Add .ZIP Library...
2. Add the TensorFlow Arduino Library .zip you downloaded.

2. LSM9DS1 Arduino Library

This library allows you to read the accelerometer, magnetometer and gyroscope values from the LSM9DS1 IMU on your Arduino Nano 33 BLE Sense.

1. From the Arduino menu, select Sketch -> Include -> Manage Libraries...
2. Search and Install "Arduino_LSM9DS1".

Note: From the Arduino menu, select File -> Examples, you should now be able to see Arduino_LSM9DS1 and Arduino_TensorFlowLite in the menu. If not, run the above instructions again and make sure you've installed them correctly.

5. Load and Build the example

1. Load the example

From the Arduino menu, select File -> Examples -> Arduino_TensorFlowLite -> magic_wand to load the sample code.

This will load the magic wand source code.

2. Build the example

Click the Upload button in the sketch window.

Note: If you see compiler errors, then something went wrong with previous software installation steps. Fix the issues based on the error you notice. Removing and reattaching the USB cable to the computer/laptop may help.

After a few minutes you should see red text indicating flashing has completed. During upload, the right LED should pulse, and then go dark at the end.

6. Demo

The magic wand can currently detect 3 gestures as shown below:

1. Wing: Start from the upper left corner and carefully trace the letter "W" for two seconds.
2. Ring: Start upright, move the wand in a clockwise circle for one second.
3. Slope: Start by holding the wand facing upward, with the LEDs toward you. Move the wand down at an incline to the left and then horizontally to the right for one second.

The following image demonstrates two gestures. First, a Slope and then a Wing (use this as a reference for your demo).

To run the demo, follow the instructions given below:

1. With the USB plugged in, select Tools -> Serial Monitor from the Arduino menu. It will initially open a blank screen with no output.

2. Move the Arduino microcontroller to trace each of the above shapes carefully and see if the serial monitor detects the gesture.

3. From the output in the serial monitor, we notice that the magic wand indeed detected all the shapes! You'll also notice that the right LED lights up.

7. Next Steps

Congratulations, you've successfully built your first gesture-recognizing "magic wand" on an Arduino microcontroller!

We hope you've enjoyed this brief introduction to development with TensorFlow Lite for Microcontrollers. The idea of deep learning on microcontrollers is new and exciting, and we encourage you to go out and experiment!

Reference docs

• Train your own model, now that you have experience working with the basic program.
• Learn more about TensorFlow Lite for Microcontrollers ( Website, GitHub).
• Try other examples and try running them on the Arduino, if it's supported.
• Try the other codelab: AI on a microcontroller with TensorFlow Lite and SparkFun Edge
• Refer to the O'Reilly book TinyML: Machine Learning with TensorFlow on Arduino and Ultra-Low Power Micro-Controllers, which introduces machine learning on tiny devices and walks through several fun projects. This codelab is based on Chapter 11 of this book.

Thanks, and have fun building!