Contact sales

Use Tensor Flow Lite and a Particle Xenon to build the ML-gesture wand of your dreams

In this Machine Learning tutorial, you'll learn how-to use a Particle Argon and a pre-trained ML model to recognize hand gestures performed in the air.

Brandon Satrom article author avatarBrandon SatromNovember 26, 2019
Use Tensor Flow Lite and a Particle Xenon to build the ML-gesture wand of your dreams

In my last post, I showed how you can get started with Machine Learning on Particle devices with our new support for TensorFlow Lite for Microcontrollers. As a part of that post, we looked at an end-to-end example of building an MCU-friendly TensorFlow Lite model, and performing real-time inferencing (aka prediction) against that model. I did so with a toy example in order to keep the focus on the process of using TensorFlow Lite in your Particle apps.

For this post, I’ll bring our ML exploration firmly into the realm of IoT and share how to use a pre-trained ML model to recognize “gestures” performed in the air by running inference against data streaming from a 3-axis accelerometer. The original source for this project is part of the official TensorFlow repository, though I made a number of modifications to support a different accelerometer breakout, and to add a bit of Particle RGB LED flair at the end. You may not actually be a magician by the end of this post, but you’ll certainly feel like one!

Parts and tools

If you want to build one of these wands for yourself, you’ll need the following components.

The Problem We’re Trying to Solve

One of the reasons that ML on MCUs is such a hot topic these days is because of the amount of data for some solutions can be a deterrent to performing inferencing in the cloud. If you’re working with a high-precision sensor and you need to make a decision based on raw data, you often cannot shoulder the cost of sending all that data to the cloud. And even if you could, the round trip latency runs counter to the frequent need for real-time decision-making.

By performing prediction on a microcontroller, you can process all that raw data on the edge, quickly, without sending a single byte to the cloud.

For this post, I’ll illustrate this with an example that works with a common IoT device: a 3-axis accelerometer, which is connected to a Particle Xenon. The Xenon will use streaming accelerometer data and TensorFlow Lite for Microcontrollers to determine if the person holding the device is making one of 3 gestures:

  • A clockwise circle, or “ring” gesture;
  • A capital W, or “wing” gesture;
  • A right to left “slope” gesture.

Here’s a GIF sample of yours truly performing these gestures.

If one of the three gestures is detected, the onboard RGB LED will light up green, blue, or red, depending on the gesture performed.

Using an Accelerometer Gesture Detection Model

For this post, I’ll be using a pre-trained model that’s included in the example projects for the TensorFlow Lite library. The model, which was created by the TensorFlow team, is a 20 KB convolutional neural network (or CNN) trained on gesture data from 10 people performing four gestures fifteen times each (ring, wing, slope, and an unknown gesture). For detection (inferencing), the model accepts raw accelerometer data in the form of 128 X, Y, and Z values and outputs probability scores for our three gestures and an unknown. The four scores sum to 1, and we’ll consider a probability of 0.8 or greater as a confident prediction of a given gesture.

The source for this post can be found in the examples folder of the TensorFlow Lite library source. The model itself is contained in the file magic_wand_model_data.cpp, which, as I discussed in my last post, is a C array representation of the TFLite flatbuffer model itself. For MCUs, we use models in this form because we don’t have a filesystem to store or load models.

Configuring the Magic Wand Application

The entry-point of my application is in the magic_wand.cpp, which contains the setup and loop functions we’re used to. Before I get there however, I’ll load some TensorFlow dependencies and configure a few variables for our application. One of note is an object to set aside some memory for TensorFlow to store input, output and intermediate arrays in.

constexpr int kTensorArenaSize = 60 * 1024;
uint8_t tensor_arena[kTensorArenaSize];

The kTensorArenaSize variable represents the number of bytes to set aside in memory for my model to use at runtime. This will vary from one model to the next, and may need to be adjusted based on the device you’re using, as well. I ran this example on a Particle Xenon with the above values, but noticed that the same value on an Argon or Boron tended to cause my application to run out of memory quickly. If you see red SOS errors from your onboard RGB LED when running a TensorFlow project, try adjusting this value first.

Once I’ve configured the memory space for my app, I’ll load my model, and configure my operations resolver. The latter step is notable because it’s a bit different than the hello_world example I used in my last post.

static tflite::MicroMutableOpResolver micro_mutable_op_resolver;
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
    tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_MAX_POOL_2D,
    tflite::ops::micro::Register_MAX_POOL_2D());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_CONV_2D,
    tflite::ops::micro::Register_CONV_2D());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_FULLY_CONNECTED,
    tflite::ops::micro::Register_FULLY_CONNECTED());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_SOFTMAX,
    tflite::ops::micro::Register_SOFTMAX());

In the hello_world example, we used the tflite::ops::micro::AllOpsResolver, which just loads all available operations into the application. For large models, free memory will be at a premium, so you’ll want to load only the operations that your model needs for inferencing. This requires an understanding of the model architecture. For the magic_wand example, I added the operations above (DEPTHWISE_CONV_2D, MAX_POOL_2D, etc.) because those reflect the layers of the CNN model I’m using for this application.

Once everything is configured on the TensorFlow side, it’s time to set-up an accelerometer and start collecting real data.

TfLiteStatus setup_status = SetupAccelerometer(error_reporter);

This function lives in the particle_accelerometer_handler.cpp file. True to its name, it configures the accelerometer I’m using for this application. When creating this example, I used the Adafruit LSM9DS1, which contains a 3-axis accelerometer, a gyroscope, and a magnetometer.

To interface with the accelerometer, I’ll use a library. There are a couple of LSM9DS1 libraries available, but this particular application requires the use of the FIFO buffer on the LSM9DS1 for storing accelerometer readings, so for this example I am using the LSM9DS1_FIFO library.

#include <LSM9DS1_FIFO.h>
#include <Adafruit_Sensor.h>

LSM9DS1_FIFO accel = LSM9DS1_FIFO();

TfLiteStatus SetupAccelerometer(tflite::ErrorReporter *error_reporter)
{
  while (!Serial) {}

  if (!accel.begin())
  {
    error_reporter->Report("Failed to initialize accelerometer. Please reset");
    return kTfLiteError;
  }

  accel.setupAccel(accel.LSM9DS1_ACCELRANGE_2G);
  accel.setupMag(accel.LSM9DS1_MAGGAIN_4GAUSS);
  accel.setupGyro(accel.LSM9DS1_GYROSCALE_245DPS);

  error_reporter->Report("Magic starts!");

  return kTfLiteOk;
}

Once I’ve installed the library, configuration is as simple as calling accel.begin(). With that done, I’m ready to start taking readings.

Reading from the accelerometer

Each time through the loop, I’ll grab data from the accelerometer by calling the ReadAccelerometer function.

bool got_data = ReadAccelerometer(error_reporter, model_input->data.f,
                                  input_length, should_clear_buffer);

This function, which can also be found in the particle_accelerometer_handler.cpp file, reads from the FIFO buffer on the accelerometer one sample at a time. It then downsamples data from 119 Hz, the rate that my accelerometer captures data, to about 25 Hz, the rate that the model was trained on. Next, I’ll assign the x, y, and z values to an array.

while (accel.accelerationAvailable())
{
  accel.read();

  sensors_event_t accelData, magData, gyroData, temp;
  // Read each sample, removing it from the device's FIFO buffer
  if (!accel.getEvent(&accelData, &magData, &gyroData, &temp))
  {
    error_reporter->Report("Failed to read data");
    break;
  }

  // Throw away this sample unless it's the nth
  if (sample_skip_counter != sample_every_n)
  {
    sample_skip_counter += 1;
    continue;
  }
    
  save_data[begin_index++] = accelData.acceleration.x * 1000;
  save_data[begin_index++] = accelData.acceleration.y * 1000;
  save_data[begin_index++] = accelData.acceleration.z * 1000;

  sample_skip_counter = 1;
  // If we reached the end of the circle buffer, reset
  if (begin_index >= 600)
  {
    begin_index = 0;
  }
  new_data = true;

  break;
}

This function will run each time through the loop, accumulating readings until we have enough data to perform a prediction, which is around 200 x, y, and z values. After I’ve collected those, I set all the values on the input tensor of my model and I’m ready to make a prediction!

for (int i = 0; i < length; ++i)
{
  int ring_array_index = begin_index + i - length;
  if (ring_array_index < 0)
  {
    ring_array_index += 600;
  }
  input[i] = save_data[ring_array_index];
}

Real-time inferencing with accelerometer data

With a set of accelerometer values in my model input, I can make a prediction by invoking the TFLite interpreter. Notice that the snippet below is identical to the hello world example from my last post. Setting inputs and reading outputs will differ from one TFLite application to the next, but the invocation process follows a consistent API.

TfLiteStatus invoke_status = interpreter->Invoke();

if (invoke_status != kTfLiteOk)
{
  error_reporter->Report("Invoke failed on index: %d\n", begin_index);
  return;
}

Once I’ve made a prediction, I work with the output and determine if a gesture was found.

int gesture_index = PredictGesture(interpreter->output(0)->data.f);

The PredictGesture function takes the values from my output tensor and determines if any value is over the 80% confidence threshold I’ve set and, if so, sets that value as the current prediction. Otherwise, an unknown value is set.

int this_predict = -1;
  for (int i = 0; i < 3; i++) { if (output[i] > 0.8) this_predict = i;
}

// No gesture was detected above the threshold
if (this_predict == -1) {
  continuous_count = 0;
  last_predict = 3;
  return 3;
}

In addition to interpreting the current prediction, we’re also tracking some state across predictions. Since accelerometer data is highly variable, we should make sure that our model predicts the same gesture multiple times in a row before we formally decide that we’re happy with the prediction. As such, the final portion of PreductGesture tracks the last and current prediction and, if the prediction has occurred a given number of times in a row, we’ll report it. Otherwise, we’ll report that the gesture was unknown. The kConsecutiveInferenceThresholds is an array of integers that correspond to the number of consecutive predictions we want to see for each gesture before considering a prediction to be valid. This may differ for your project and accelerometer. The values I chose can be found in the particle_constants.cpp file.

if (last_predict == this_predict) {
  continuous_count += 1;
} else {
  continuous_count = 0;
}

last_predict = this_predict;

if (continuous_count < kConsecutiveInferenceThresholds[this_predict]) {
  return 3;
}

continuous_count = 0;
last_predict = -1;
return this_predict;

Now that we have a result, we can do something with it.

Displaying the result

At the end of my loop, with a predicted gesture in hand, I call a function called `HandleOutput` to display something to the user. This function is defined in the particle_output_handler.cpp file and does three things:

  • Toggles the D7 LED each time an inference is performed;
  • Outputs the predicted gesture to the Serial console;
  • Sets the onboard RGB Led green, blue, or red, depending on the predicted gesture.

The first step is pretty straightforward, so I won’t cover it here. For the second, we’ll print a big of ASCII art to the console to represent each gesture.

if (kind == 0)
{
  error_reporter->Report(
      "WING:\n\r*         *         *\n\r *       * *       "
      "*\n\r  *     *   *     *\n\r   *   *     *   *\n\r    * *       "
      "* *\n\r     *         *\n\r");
}
else if (kind == 1)
{
  error_reporter->Report(
      "RING:\n\r          *\n\r       *     *\n\r     *         *\n\r "
      "   *           *\n\r     *         *\n\r       *     *\n\r      "
      "    *\n\r");
}
else if (kind == 2)
{
  error_reporter->Report(
      "SLOPE:\n\r        *\n\r       *\n\r      *\n\r     *\n\r    "
      "*\n\r   *\n\r  *\n\r * * * * * * * *\n\r");
}

It doesn’t look like much here, but if you run the example and perform a gesture, you’ll see the art in all its glory.

For the final step, we’ll do something unique to Particle devices: take control of the onboard RGB LED. The onboard LED is used by Particle to communicate device status, connectivity, and the like, but it’s also possible to take control of the LED and use it in your own applications. You simply call RBG.control(true), and then RGB.color to set an RGB value. For this example, I modified the ASCII art snippet above to set my LED, and release control with RGB.control(false) if a gesture isn’t detected.

if (kind == 0)
{
  RGB.control(true);
  RGB.color(0, 255, 0); // Green

  error_reporter-&gt;Report(
    "WING:\n\r* * *\n\r * * * "
    "*\n\r * * * *\n\r * * * *\n\r * * "
    "* *\n\r * *\n\r");
}
else if (kind == 1)
{
  RGB.control(true);
  RGB.color(0, 0, 255); // Blue

  error_reporter-&gt;Report(
    "RING:\n\r *\n\r * *\n\r * *\n\r "
    " * *\n\r * *\n\r * *\n\r "
    " *\n\r");
}
else if (kind == 2)
{
  RGB.control(true);
  RGB.color(255, 0, 0); // Red

  error_reporter-&gt;Report(
    "SLOPE:\n\r *\n\r *\n\r *\n\r *\n\r "
    "*\n\r *\n\r *\n\r * * * * * * * *\n\r");
}
else
{
  RGB.control(false);
}

 

Now, in addition to fancy ASCII art, your device will change the RGB LED when a gesture is detected.

Taking your exploration further

In this post, we took our exploration of TensorFlow Lite into the realm of IoT with an accelerometer demo. Now it’s your turn. If you haven’t already, check out the TensorFlowLite Particle library and explore some of the other demos. And if you’re building your own ML on MCUs project with TensorFlow, share it in the comments below!

Related Articles