Machine learning helps developers build software that can understand our world. We can use it to create intelligent tools that make users' lives easier, like the Google Assistant, and fun experiences that let users express their creativity, like Google Pixel's portrait mode.
But often, these experiences require a lot of computation. Machine learning often needs to run on a powerful cloud server, or at least a powerful mobile phone.
With TensorFlow Lite, it's possible to run machine learning inference on tiny, low-powered hardware, like microcontrollers. This means you can build amazing experiences that add intelligence to the smallest devices, bringing machine learning closer to the world around us.
In this codelab, we'll learn to deploy a machine learning model to the SparkFun TensorFlow, a microcontroller designed by Google and SparkFun to help developers experiment with ML on tiny devices.
The SparkFun TensorFlow is a microcontroller: a tiny computer on a single circuit board. It has a processor, memory, and I/O hardware that allows it to send and receive digital signals to other devices. It also has four software-controllable LEDs, in your favorite Google colors.
Unlike a computer, a microcontroller doesn't usually run an operating system. Instead, the programs you write run directly on the hardware. You write your code on a computer and download it to the microcontroller via a device called a programmer. We'll hear more about those later.
Microcontrollers are not powerful computers. They have small processors, and not much memory. But because they are designed to be as simple as possible, a microcontroller can use very little energy. Depending on what your program does, the SparkFun TensorFlow can run for weeks on a single coin cell battery!
TensorFlow is Google's framework for building and training machine learning models, and TensorFlow Lite is a set of tools for running those models on small, relatively low-powered devices. This could mean mobile phones, and it could mean microcontrollers like the SparkFun TensorFlow.
Microcontrollers are extremely common. They are part of all sorts of household devices: think appliances, cars, and toys. In fact, there are around 30 billion microcontroller-powered devices produced each year. They're cheap, require very little energy, and are very reliable.
By bringing machine learning to tiny microcontrollers, we can boost the intelligence of billions of devices that we use in our lives, without relying on expensive hardware or reliable internet connections. Imagine smart appliances that can adapt to your daily routine, intelligent industrial sensors that understand the difference between problems and normal operation, and magical toys that can help kids learn in fun and delightful ways.
For this codelab we'll be working with the board's built-in speech detection model, which uses a convolutional neural network to detect the words "yes" and "no" being spoken via the board's two microphones.
This codelab is designed to get you started compiling code and deploying it to your device.
You will need the following hardware:
You'll also need the following software:
python --versionat the command line)
First, let's power up your board. To do this, insert a coin cell battery into the battery connector on the back of the board, with the "+" side of the battery facing up. If your board came with a battery already inserted, pull out the plastic tab, and push the battery to ensure it is fully inserted.
Once you've inserted the battery, the board will wake up and begin listening with its microphones. The blue light should begin to flash. It flashes once for each
The machine learning model on the board is trained to recognize the words "yes" and "no", and to detect the presence and absence of speech. It communicates its results by lighting colored LEDs. The following table shows the meaning of each LED color:
No speech detected
No LEDs lit
Try holding the board up to your face and saying "yes" a few times. You should see the yellow LED light up.
If nothing happens when you say "yes", here are some things to try:
Now that you've tested your board's built-in speech detection model, we're going to download the source code for this program and build it. The program is written in C++, so it must be compiled into a binary before being downloaded onto the board. A binary is a file that contains the program in a form that can be run directly by the SparkFun TensorFlow hardware.
The following instructions are written for MacOS or Linux. If you're using Windows, skip ahead to the Set up your software (Windows) section.
The code is available in the TensorFlow repository on GitHub, in the following location:
To download the code, open a terminal on your computer and change to a directory where you usually store coding projects. If you can't think of a good place, change to your home directory by entering
Once you're in the appropriate directory, enter the following command to download the TensorFlow repository:
git clone https://github.com/tensorflow/tensorflow.git
Next, enter the directory that was created in the previous step.
Run the following script to download the dependencies we'll need to build this project.
The script will take a while to run. You'll eventually see the following:
download_dependencies.sh completed successfully.
Next, run the following command to build and test TensorFlow Lite's microcontroller library. This will help us know we have the correct dependencies.
make -f tensorflow/lite/experimental/micro/tools/make/Makefile test
When the library has finished building and the tests have successfully run, you should see the following output:
~~~ALL TESTS PASSED~~~
We'll be using Python 3 to prepare our binary and flash it to the device. The Python scripts depend on certain libraries being available.
Run the following command to use
pip to install these dependencies:
pip3 install pycrypto pyserial
The SparkFun TensorFlow stores the program it is currently running in its 512 kilobytes of flash memory. If we want the board to run a new program, we have to send it to the board, which will store it in flash memory, overwriting any program that was previously saved.
This process is called "flashing", and we'll use it to send our program to the board. But first we need to build the binary!
First, if you're using MacOS, we'll need to run the following command. If you're using Linux, you don't need to do this.
Next, initiate the build process, issue the following command.
make -f tensorflow/lite/experimental/micro/tools/make/Makefile TARGET=sparkfun_edge micro_speech_bin
If the build works successfully, the final line of the output should appear as follows:
arm-none-eabi-objcopy tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin -O binary
To confirm that the binary was successfully created, run the following command:
test -f tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin; if [ $? -eq 0 ]; then echo "Binary was successfully created"; else echo "Binary is missing"; fi
You should see
Binary was successfully created printed to the console!
If you see
Binary is missing, there was a problem with the build process that will require debugging.
First, we'll run a script that will adapt our binary so it can be downloaded to the SparkFun TensorFlow.
Enter the following commands:
cp tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/keys_info0.py tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/keys_info.py python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/create_cust_image_blob.py --bin tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin --load-address 0xC000 --magic-num 0xCB -o main_nonsecure_ota --version 0x0
This will create the file
main_nonsecure_ota.bin. We'll now run another command to create a final version of the file:
python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/create_cust_wireupdate_blob.py --load-address 0x20000 --bin main_nonsecure_ota.bin -i 6 -o main_nonsecure_wire --options 0x1
You should now have a file called
main_nonsecure_wire.bin. This is the file we'll be flashing to the device.
To download new programs to the board, we'll be using the SparkFun USB-C Serial Basic serial programmer. This device allows your computer to communicate with the microcontroller via USB.
To attach this device to your board, perform the following steps:
We'll be connecting the board to your computer via USB. To program the board, we'll need to know the name that your computer gives the device. The best way of doing this is to list all the computer's devices before and after attaching it, and look to see which device is new.
Before attaching the device via USB, run the following command:
If you are using MacOS: ls /dev/cu* If you are using Linux: ls /dev/tty*
This should output a list of attached devices that looks something like the following:
/dev/cu.Bluetooth-Incoming-Port /dev/cu.MALS /dev/cu.SOC
Now, connect the programmer to your computer's USB port. Enter the following command again:
If you are using MacOS: ls /dev/cu* If you are using Linux: ls /dev/tty*
You should see an extra item in the output, as in the example below. Your new item may have a different name. This new item is the name of the device.
/dev/cu.Bluetooth-Incoming-Port /dev/cu.MALS /dev/cu.SOC /dev/cu.usbserial-1450
This name is used to refer to the device. However, it may change depending on which USB port the programmer is attached to, so if you remove and reattach the USB from the computer then you may have to look up its name again.
The script we'll be using to flash the board has a minor issue that can be fixed by adding two lines of code.
With a text editor, open the following file:
Locate the code around line
35, which looks like the following:
with serial.Serial(args.port, args.baud, timeout=12) as ser: connect_device(ser)
Update the code so it looks exactly as follows, making sure the indentation is correct:
with serial.Serial(args.port, args.baud, timeout=12) as ser: import time time.sleep(0.25) connect_device(ser)
Finally, save the file!
To flash the board, we have to put it into a special "bootloader" state that prepares it to receive the new binary. We'll then run a script to send the binary to the board.
First, paste the following command into your terminal, substituting
DEVICENAME with your programmer device's name from the previous sections. However, do not hit enter yet!
python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py -b 115200 DEVICENAME -r 1 -f main_nonsecure_wire.bin -i 6
Now, we'll reset the board into its bootloader state and flash the board!
You'll need to be familiar with the following buttons:
Perform the following steps:
14. Keep holding it!
14, click the button marked
RSTto reset the board.
You should now see something like the following appearing on-screen:
Connecting with Corvette over serial port /dev/cu.usbserial-1440... Sending Hello. Received response for Hello Received Status length = 0x58 version = 0x3 Max Storage = 0x4ffa0 Status = 0x2 State = 0x7 AMInfo = 0x1 0xff2da3ff 0x55fff 0x1 0x49f40003 0xffffffff [...lots more 0xffffffff...] Sending OTA Descriptor = 0xfe000 Sending Update Command. number of updates needed = 1 Sending block of size 0x158b0 from 0x0 to 0x158b0 Sending Data Packet of length 8180 Sending Data Packet of length 8180 [...lots more Sending Data Packet of length 8180...]
Keep holding button 14 until you see
Sending Data Packet of length 8180!
The program will continue to print lines on the terminal. It will eventually look something like the following:
[...lots more Sending Data Packet of length 8180...] Sending Data Packet of length 8180 Sending Data Packet of length 6440 Sending Reset Command. No response for command 0x00000006 received bytes 19 ['0xb7', '0xa5', '0x61', '0x81', '0x7', '0x0', '0x14', '0x0', '0x6', '0x0', '0x0', '0x0', '0x0', '0x0', '0x0', '0x0', '0x0', '0x0', '0x0'] Traceback (most recent call last): File "tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py", line 336, in <module> main() File "tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py", line 38, in main connect_device(ser) File "tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py", line 172, in connect_device send_ackd_command(resetmsg, ser) File "tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py", line 191, in send_ackd_command response = send_command(command, 20, ser) File "tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py", line 235, in send_command raise NoResponseError __main__.NoResponseError
Despite the error message, this generally indicates a successful flashing!
Once your board has been successfully flashed, it will reset. The program we've just flashed onto the board will blink the blue LED, so if the blue LED starts blinking, flashing was successful.
To confirm this, test the board by saying "yes" until the yellow LED lights up!
Here are some possible issues and how to debug them:
Problem: The flashing script hangs for a while at "Sending Hello." and then prints an error.
Solution: You need to hold down the button marked
14 while running the script. Hold down button
14, hit the
RST button, then run the script, while holding the button marked
14 the whole time.
Problem: After flashing, none of the LEDs are coming on.
Solution: Try hitting the
RST button, disconnecting the board from the programmer, and removing and replacing the battery. If none of these work, try flashing the board again.
Problem: The blue LED is lighting up, but it's very dim.
Solution: The battery is running low and needs to be replaced. The board can also be powered by the programmer, so if your battery has run out, just keep it plugged in to the programmer and use it as normal.
Now that you know how to build and flash your SparkFun TensorFlow, you can start playing with the code and deploying it to your device to see the results.
A good place to start reading the code is the following file:
You can see the file on GitHub here.
The method in this file is called when a voice command is detected. Try modifying the behavior - can you make a different LED turn on?
Once you've made code changes, follow the steps earlier in the codelab to build the project, prepare the binary, and flash it to your device.
We hope you've enjoyed this brief introduction to development with the SparkFun TensorFlow!
Here are some good places to learn more: