Machine learning helps developers build software that can understand our world. We can use it to create intelligent tools that make users' lives easier, like the Google Assistant, and fun experiences that let users express their creativity, like Google Pixel's portrait mode.

But often, these experiences require a lot of computation. Machine learning often needs to run on a powerful cloud server, or at least a powerful mobile phone.

With TensorFlow Lite, it's possible to run machine learning inference on tiny, low-powered hardware, like microcontrollers. This means you can build amazing experiences that add intelligence to the smallest devices, bringing machine learning closer to the world around us.

In this codelab, we'll learn to deploy a machine learning model to the SparkFun Edge, a microcontroller designed by Google and SparkFun to help developers experiment with ML on tiny devices.

What is the SparkFun Edge?

The SparkFun Edge is a microcontroller-based platform: a tiny computer on a single circuit board. It has a processor, memory, and I/O hardware that allows it to send and receive digital signals to other devices. It also has four software-controllable LEDs, in your favorite Google colors.

Unlike a computer, a microcontroller doesn't usually run an operating system. Instead, the programs you write run directly on the hardware. You write your code on a computer and download it to the microcontroller via a device called a programmer. We'll hear more about those later.

Microcontrollers are not powerful computers. They have small processors, and not much memory. But because they are designed to be as simple as possible, a microcontroller can use very little energy. Depending on what your program does, the SparkFun Edge can run for weeks on a single coin cell battery!

What is TensorFlow Lite, and why do ML on a tiny device?

TensorFlow is Google's framework for building and training machine learning models, and TensorFlow Lite is a set of tools for running those models on small, relatively low-powered devices. This could mean mobile phones, and it could mean microcontrollers like the SparkFun Edge.

Microcontrollers are extremely common. They are part of all sorts of household devices: think appliances, cars, and toys. In fact, there are around 30 billion microcontroller-powered devices produced each year. They're cheap, require very little energy, and are very reliable.

By bringing machine learning to tiny microcontrollers, we can boost the intelligence of billions of devices that we use in our lives, without relying on expensive hardware or reliable internet connections. Imagine smart appliances that can adapt to your daily routine, intelligent industrial sensors that understand the difference between problems and normal operation, and magical toys that can help kids learn in fun and delightful ways.

What you'll build

For this codelab we'll be working with the board's built-in speech detection model, which uses a convolutional neural network to detect the words "yes" and "no" being spoken via the board's two microphones.

This codelab is designed to get you started compiling your own code and deploying it to your device.

What you'll learn

What you'll need

You will need the following hardware:

You'll also need the following software:

First, let's power up your board. To do this, insert a coin cell battery into the battery connector on the back of the board, with the "+" side of the battery facing up. If your board came with a battery already inserted, pull out the plastic tab, and push the battery to ensure it is fully inserted.

If you don't have a coin battery, you can use the SparkFun USB-C Serial Basic programmer device to power the board.

To attach this device to your board, perform the following steps:

  1. Locate the six pin header on the side of the SparkFun Edge.
  2. Plug the SparkFun USB-C Serial Basic into these pins, ensuring the pins labelled "BLK" and "GRN" on each device are lined up correctly.
  3. Connect a USB-C cable between the SparkFun USB-C Serial Basic and your computer.

Once you've inserted the battery or connected the USB programmer, the board will wake up and begin listening with its microphones. The blue light should begin to flash.

The machine learning model on the board is trained to recognize the words "yes" and "no", and to detect the presence and absence of speech. It communicates its results by lighting colored LEDs. The following table shows the meaning of each LED color:

Detection result

LED color

"Yes"

Yellow

"No"

Red

Unknown speech

Green

No speech detected

Flashing blue LED only

Give it a try

Try holding the board up to your face and saying "yes" a few times. You should see the yellow LED light up.

If nothing happens when you say "yes", here are some things to try:

We're now going to download the source code for this program and the dependencies we need to build it. The program is written in C++, which must be compiled into a binary before being downloaded onto the board. A binary is a file that contains the program in a form that can be run directly by the SparkFun Edge hardware.

The following instructions are written for MacOS or Linux.

Download the TensorFlow repo

The code is available in the TensorFlow repository on GitHub, in the following location:

https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/micro

To download the code, open a terminal on your computer and change to a directory where you usually store coding projects. If you can't think of a good place, change to your home directory by entering cd ~.

Once you're in the appropriate directory, enter the following command to download the TensorFlow repository:

curl -o tf.zip https://codeload.github.com/tensorflow/tensorflow/zip/aa47072ff4e2b7735b0e0ef9ef52f68ffbf7ef54
unzip tf.zip

Next, enter the directory that was created in the previous step.

cd tensorflow-aa47072ff4e2b7735b0e0ef9ef52f68ffbf7ef54

Download Python dependencies

We'll be using Python 3 to prepare our binary and flash it to the device. The Python scripts depend on certain libraries being available.

Run the following command to use pip to install these dependencies:

pip3 install pycrypto pyserial --user

Next, we're going to build the binary and run some commands that get it ready for downloading to the device.

Build the binary

To download all of the required dependencies and initiate the build process, issue the following command.

make -f tensorflow/lite/experimental/micro/tools/make/Makefile \
TARGET=sparkfun_edge micro_speech_bin

If the build works successfully, the final line of the output should appear as follows:

arm-none-eabi-objcopy tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin -O binary

To confirm that the binary was successfully created, run the following command:

test -f tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin &&  echo "Binary was successfully created" || echo "Binary is missing"

You should see Binary was successfully created printed to the console!

If you see Binary is missing, there was a problem with the build process that will require debugging.

Prepare the binary

The binary must be signed with cryptographic keys to be deployed to the device. We'll now run some commands that will sign our binary so it can be downloaded to the SparkFun Edge.

Enter the following command to set up some dummy cryptographic keys we can use for development:

cp tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/keys_info0.py \
tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/keys_info.py

Now, run the following command to create a signed binary:

python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/create_cust_image_blob.py \
--bin tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin \
--load-address 0xC000 \
--magic-num 0xCB \
-o main_nonsecure_ota \
--version 0x0

This will create the file main_nonsecure_ota.bin. We'll now run another command to create a final version of the file that can be used to flash our device with the bootloader script we will use in the next step:

python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/create_cust_wireupdate_blob.py \
--load-address 0x20000 \
--bin main_nonsecure_ota.bin \
-i 6 \
-o main_nonsecure_wire \
--options 0x1

You should now have a file called main_nonsecure_wire.bin in the directory where you ran the commands. This is the file we'll be flashing to the device.

What is flashing?

The SparkFun Edge stores the program it is currently running in its 512 kilobytes of flash memory. If we want the board to run a new program, we have to send it to the board, which will store it in flash memory, overwriting any program that was previously saved.

This process is called "flashing", and we'll use it to send our program to the board.

Attach the programmer to the board

To download new programs to the board, we'll be using the SparkFun USB-C Serial Basic serial programmer. This device allows your computer to communicate with the microcontroller via USB.

To attach this device to your board, perform the following steps:

  1. Locate the six pin header on the side of the SparkFun Edge.
  2. Plug the SparkFun USB-C Serial Basic into these pins, ensuring the pins labelled "BLK" and "GRN" on each device are lined up correctly.

Attach the programmer to your computer

We'll be connecting the board to your computer via USB. To program the board, we'll need to know the name that your computer gives the device. The best way of doing this is to list all the computer's devices before and after attaching it, and look to see which device is new.

Before attaching the device via USB, run the following command:

If you are using MacOS: ls /dev/cu*
If you are using Linux: ls /dev/tty*

This should output a list of attached devices that looks something like the following:

/dev/cu.Bluetooth-Incoming-Port
/dev/cu.MALS
/dev/cu.SOC

Now, connect the programmer to your computer's USB port. Enter the following command again:

If you are using MacOS: ls /dev/cu*
If you are using Linux: ls /dev/tty*

You should see an extra item in the output, as in the example below. Your new item may have a different name. This new item is the name of the device.

/dev/cu.Bluetooth-Incoming-Port
/dev/cu.MALS
/dev/cu.SOC
/dev/cu.wchusbserial-1450

This name is used to refer to the device. However, it may change depending on which USB port the programmer is attached to, so if you remove and reattach the USB from the computer then you may have to look up its name again.

Once you've identified the device name, put it in a shell variable for later use:

export DEVICENAME=put your device name here

Run the script to flash your board

To flash the board, we have to put it into a special "bootloader" state that prepares it to receive the new binary. We'll then run a script to send the binary to the board.

First, we'll create an environment variable to specify the baud rate, which is the speed at which data will be sent to the device:

export BAUD_RATE=921600

Now paste the following command into your terminal. The `${DEVICENAME}` and `${BAUD_RATE}` in the command will be replaced with the values you set in the previous sections.

Do not hit enter yet!

python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py -b ${BAUD_RATE} ${DEVICENAME} -r 1 -f main_nonsecure_wire.bin -i 6

Now, we'll reset the board into its bootloader state and flash the board!

You'll need to be familiar with the following buttons:

Perform the following steps:

  1. Ensure your board is connected to the programmer, and the entire thing is connected to your computer via USB.
  2. On the board, hold the button marked 14. Keep holding it!
  3. Still holding the button marked 14, click the button marked RST to reset the board.
  4. Hit enter on your computer to run the script. Keep on holding button 14!

You should now see something like the following appearing on-screen:

Connecting with Corvette over serial port /dev/cu.usbserial-1440...
Sending Hello.
Received response for Hello
Received Status
length =  0x58
version =  0x3
Max Storage =  0x4ffa0
Status =  0x2
State =  0x7
AMInfo =
0x1
0xff2da3ff
0x55fff
0x1
0x49f40003
0xffffffff
[...lots more 0xffffffff...]
Sending OTA Descriptor =  0xfe000
Sending Update Command.
number of updates needed =  1
Sending block of size  0x158b0  from  0x0  to  0x158b0
Sending Data Packet of length  8180
Sending Data Packet of length  8180
[...lots more Sending Data Packet of length  8180...]

Keep holding button 14 until you see Sending Data Packet of length 8180. You can release the button after seeing this (but it's okay if you keep holding it).

The program will continue to print lines on the terminal. It will eventually look something like the following:

[...lots more Sending Data Packet of length  8180...]
Sending Data Packet of length  8180
Sending Data Packet of length  6440
Sending Reset Command.
Done.

This indicates a successful flashing!

Try the program out

Once your board has been successfully flashed, it will reset. The program we've just flashed onto the board will blink the blue LED, so if the blue LED starts blinking, flashing was successful.

To confirm this, test the board by saying "yes" until the yellow LED lights up!

What if it didn't work?

Here are some possible issues and how to debug them:

Problem: The flashing script hangs for a while at "Sending Hello." and then prints an error.

Solution: You need to hold down the button marked 14 while running the script. Hold down button 14, hit the RST button, then run the script, while holding the button marked 14 the whole time.

Problem: After flashing, none of the LEDs are coming on.

Solution: Try hitting the RST button, disconnecting the board from the programmer, and removing and replacing the battery. If none of these work, try flashing the board again.

Problem: The blue LED is lighting up, but it's very dim.

Solution: The battery is running low and needs to be replaced. The board can also be powered by the programmer, so if your battery has run out, just keep it plugged in to the programmer and use it as normal.

Now that you know how to build and flash your SparkFun Edge, you can start playing with the code and deploying it to your device to see the results.

Read the code

A good place to start reading the code is the following file, command_responder.cc.

tensorflow/lite/experimental/micro/examples/micro_speech/sparkfun_edge/command_responder.cc

You can see the file on GitHub here.

The method in this file, RespondToCommand, is called when a voice command is detected. The existing code turns on a different LED depending on whether "yes", "no", or an unknown command was heard. The following snippet shows how this works:

if (found_command[0] == 'y') {
  am_hal_gpio_output_set(AM_BSP_GPIO_LED_YELLOW);
}
if (found_command[0] == 'n') {
  am_hal_gpio_output_set(AM_BSP_GPIO_LED_RED);
}
if (found_command[0] == 'u') {
  am_hal_gpio_output_set(AM_BSP_GPIO_LED_GREEN);
}

The found_command argument contains the name of the command that was detected. By checking the first character, this set of if statements determines which LED to light.

The method RespondToCommand is called with several arguments:

void RespondToCommand(tflite::ErrorReporter* error_reporter,
    int32_t current_time, const char* found_command,
    uint8_t score, bool is_new_command) {

The score is an integer number from 0-255 that represents the probability that a command was detected. The sample code only considers a command as valid if the score is greater than 200. Based on our testing, most valid commands fall within the range of 200-210.

Modify the code


The SparkFun Edge board has four LEDs. Currently, we're flashing the blue LED to indicate that recognition is occurring. You can see this in the command_responder.cc file:

static int count = 0;

// Toggle the blue LED every time an inference is performed.
++count;
if (count & 1) {
  am_hal_gpio_output_set(AM_BSP_GPIO_LED_BLUE);
} else {
  am_hal_gpio_output_clear(AM_BSP_GPIO_LED_BLUE);
}

Since we have a bank of four LEDs, let's modify the program to use them as a visual indicator of the score of a given command. A low score will merit a single lit LED, and a high score will result in multiple lights.

To ensure we have a way to know that the program is running, we'll make the red LED flash continually instead of the blue. The adjacent blue, green, and yellow LEDs will be used to show the strength of our most recent score. And for simplicity, we'll only light up those LEDs if the word "yes" is spoken. If another word is detected, the LEDs will clear.

To make this change, replace all the code in your command_responder.cc file with the following snippet:

#include "tensorflow/lite/experimental/micro/examples/micro_speech/command_responder.h"

#include "am_bsp.h"

// This implementation will light up the LEDs on the board in response to
// different commands.
void RespondToCommand(tflite::ErrorReporter* error_reporter,
                      int32_t current_time, const char* found_command,
                      uint8_t score, bool is_new_command) {
  static bool is_initialized = false;
  if (!is_initialized) {
    // Setup LEDs as outputs
    am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_RED, g_AM_HAL_GPIO_OUTPUT_12);
    am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_BLUE, g_AM_HAL_GPIO_OUTPUT_12);
    am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_GREEN, g_AM_HAL_GPIO_OUTPUT_12);
    am_hal_gpio_pinconfig(AM_BSP_GPIO_LED_YELLOW, g_AM_HAL_GPIO_OUTPUT_12);
    // Ensure all pins are cleared
    am_hal_gpio_output_clear(AM_BSP_GPIO_LED_RED);
    am_hal_gpio_output_clear(AM_BSP_GPIO_LED_BLUE);
    am_hal_gpio_output_clear(AM_BSP_GPIO_LED_GREEN);
    am_hal_gpio_output_clear(AM_BSP_GPIO_LED_YELLOW);
    is_initialized = true;
  }
  static int count = 0;

   // Toggle the red LED every time an inference is performed.
   ++count;
   if (count & 1) {
     am_hal_gpio_output_set(AM_BSP_GPIO_LED_RED);
   } else {
     am_hal_gpio_output_clear(AM_BSP_GPIO_LED_RED);
   }

  if (is_new_command) {
    // Clear the last three LEDs
    am_hal_gpio_output_clear(AM_BSP_GPIO_LED_BLUE);
    am_hal_gpio_output_clear(AM_BSP_GPIO_LED_GREEN);
    am_hal_gpio_output_clear(AM_BSP_GPIO_LED_YELLOW);
    error_reporter->Report("Heard %s (%d) @%dms", found_command, score,
                           current_time);
    // Only indicate a 'yes'
    if (found_command[0] == 'y') {
      // Always light the blue LED
      am_hal_gpio_output_set(AM_BSP_GPIO_LED_BLUE);
      // Light the other LEDs depending on score
      if (score >= 205) {
        am_hal_gpio_output_set(AM_BSP_GPIO_LED_GREEN);
      }
      if(score >= 210) {
        am_hal_gpio_output_set(AM_BSP_GPIO_LED_YELLOW);
      }
    }
  }
}

If a new command is detected, is_new_command will be true. We'll clear the blue, green, and yellow LEDs, then light them up again depending on the values of found_command and score.

Rebuild and flash

Once you've made code changes, use the following commands to build the project and prepare the binary.

make -f tensorflow/lite/experimental/micro/tools/make/Makefile TARGET=sparkfun_edge micro_speech_bin

python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/create_cust_image_blob.py --bin tensorflow/lite/experimental/micro/tools/make/gen/sparkfun_edge_cortex-m4/bin/micro_speech.bin --load-address 0xC000 --magic-num 0xCB -o main_nonsecure_ota --version 0x0

python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/create_cust_wireupdate_blob.py --load-address 0x20000 --bin main_nonsecure_ota.bin -i 6 -o main_nonsecure_wire --options 0x1

Next, paste the following command into your terminal.

Do not hit enter yet!

python3 tensorflow/lite/experimental/micro/tools/make/downloads/AmbiqSuite-Rel2.0.0/tools/apollo3_scripts/uart_wired_update.py -b ${BAUD_RATE} ${DEVICENAME} -r 1 -f main_nonsecure_wire.bin -i 6

Finally, to flash the board, hold the button marked 14. While continuing to hold button 14, click the button marked RST. Still holding button 14, hit enter to run the script. Do not let go of the button until flashing has started. For more detailed instructions, see the previous section.

Test on your device

Nice work! Now you've flashed the new code, you should see the red LED flashing. Try saying "yes" and see how many colored LEDs light up.

As before, the board doesn't recognize your command immediately, try repeating yourself: say "yes yes yes". Remember, ambient sounds can interfere with the accuracy of detection.

When developing software, it's important to be able to print debugging information so you can better understand what is going on when your code runs.

When working with microcontrollers, a typical way to print debug information is through the board's serial connection. You use your computer to connect to the board and display the data that the board is sending.

Open a serial connection

By default, our SparkFun Edge sample code logs any spoken commands, along with their confidence. To see the board's output you can run the following command:

screen ${DEVICENAME} 115200

You should initially see an output that looks something like the following:

Apollo3 Burst Mode is Available

                               Apollo3 operating in Burst Mode (96MHz)

Try issuing some commands by saying "yes" or "no". You should see the board printing debug information for each command:

 Heard yes (202) @65536ms

In the above log, yes refers to the command. The number 202 refers to the level of confidence that the command was heard (with 200 being the minimum). Finally, 65536ms refers to the amount of time that has elapsed since the microcontroller was last reset.

To stop viewing the debug output, hit Ctrl+A, immediately followed by the K key, then hit the Y key.

Write debug logs

You can see the code that logs this information in the command_responder.cc file you were just working with:

tensorflow/lite/experimental/micro/examples/micro_speech/sparkfun_edge/command_responder.cc

To log data, you can call the error_reporter->Report() method. It supports the standard printf tokens for string interpolation, which you can use to include important information in your logs:

error_reporter->Report("Heard %s (%d) @%dms", found_command, score,
                           current_time);

This method should come in handy when you are making your own changes to the code.

We hope you've enjoyed this brief introduction to development with TensorFlow Lite for Microcontrollers. The idea of machine learning on microcontrollers is new and exciting, and we encourage you to go out and experiment!

Now that you have experience working with the basic program, you could try retraining the model to understand different commands. Take a look at the training documentation to learn more.

Here are some suggestions for further reading:

Thanks, and have fun building!