Contact sales

How to build a gesture-based window shade remote with Particle and Edge Impulse

Use Edge Impulse and Particle to create a gesture-controlled window shade.

Olivier Bloch article author avatarOlivier BlochFebruary 07, 2025
How to build a gesture-based window shade remote with Particle and Edge Impulse

How complex can it be to run artificial intelligence (AI) on a microcontroller-powered device? 

Very! 

Creating and training Machine Learning Models is already a complex task on its own, but when you want to optimize, compile and deploy these models on constrained devices you are adding another set of complexity layers. 

Both Edge Impulse and Particle promise to make complex technologies, respectively Edge AI and IoT, more accessible and streamlined while ensuring security, scalability, and suitability for industrial and enterprise integration and deployments. 

Common Edge AI development challenges 

Before jumping in, let’s look at what developing and running ML models at the edge entails. The endeavor presents considerable challenges due to the intrinsic complexity of ML and Embedded as well as the heterogeneity of the micro-controllers’ landscape. 

Here are the most challenging tasks you must go through: 

Collecting real data from sensors to train ML models on constrained devices can be challenging due to several factors: 

  • Data Quality: Ensuring the data collected is accurate and representative of real-world scenarios is crucial. Poor quality data can lead to inaccurate models. 
  • Data Volume: Constrained devices often have limited storage capacity, making it difficult to locally collect large volumes of data necessary for training robust models. 
  • Data Transmission: Transmitting large amounts of training data from sensors from a constrained device to a Cloud Service can prove costly if done through a regular device connection to an IoT Cloud solution. 

Optimizing Machine Learning Models for constrained devices (usually with a few MB of RAM and Flash and no GPU) involves several strategies: 

  • Model Compression: Techniques such as quantization (reducing the precision of model weights) and pruning (removing redundant neurons) help reduce model size without significantly impacting accuracy. 
  • Efficient Architectures: Choosing lightweight model architectures that are designed for low-power devices, such as MobileNet or SqueezeNet, can help maintain performance while minimizing resource usage. 
  • Hardware-Specific Optimizations: Leveraging hardware-specific libraries and optimizations, such as ARM’s CMSIS-NN, can significantly enhance model performance on constrained devices. 

Compiling ML Models for Embedded Applications involves several steps: 

  • Model Conversion: Converting the trained model into a format compatible with embedded devices, such as TensorFlow Lite or ONNX. 
  • Cross-Compilation: Ensuring the model can be compiled and executed on the target device’s architecture, which may involve using specialized compilers and toolchains. 
  • Optimization: Applying further optimizations to the compiled model to ensure it runs efficiently on the embedded hardware, including reducing latency and memory usage. 

The iterative process of tuning ML models on constrained devices involves several repetitive steps: 

  • Compile: Convert the model to a format suitable for the target device. 
  • Deploy: Load the model onto the device and integrate it with the application. 
  • Test: Evaluate the model’s performance in real-world scenarios. 
  • Capture new Data: Collect new data from the device’s sensors to improve the model. 
  • Retrain: Update the model with the new data and retrain it to enhance accuracy. 
  • Adapt Code: Modify the application code to accommodate any changes in the model. 
  • Recompile: Compile the updated model and deploy it again. 
  • Test Again: Re-evaluate the model’s performance and iterate as necessary. 

The good news 

While the list of challenges above looks dauting, fear not: there are tools and solutions that abstract and simplify most of it. Edge Impulse and Particle are leading the charge to making all this accessible to more developers and usable for building actual products. Edge Impulse focuses on making ML models simple to build and train as well as tiny and deployable on constrained devices, while Particle focuses on democratizing embedded development and IoT devices connectivity and management at scale. Let’s see what the end-to-end experience looks like when combining them. 

This exploration aims at going a little beyond pre-made samples and templates, evaluating the experience when stepping outside the groomed trails of getting-started examples. Specifically, we will build a gesture recognition model used to control window shades by rotating a motion sensor along different axes, illustrating a real-life scenario requiring interoperability and integration with existing third-party solutions (in our case, a home automation application). 

TL;DR: Overall, the experience was great. Despite encountering a few rough edges, this project showcases how the Edge Impulse and Particle integration offers a robust solution for deploying Edge AI on microcontroller-powered devices without the need to be a data scientist or AI expert, nor having to go too deep in hardware, firmware and embedded development. 

Below, we detail the steps undertaken during the experiment so you can give it a try yourself or be inspired to build your next Edge AI powered solution. 

 

Building a gesture-based remote for window shades 

Prerequisites 

If you want to reproduce the below experiment, you will need the following: 

 

In terms of development tools, you will need: 

Last but not least, you will need a Particle and an Edge Impulse account. 

Hello Muon world 

To make sure you have a working dev environment, it is always good to go through some hello world sample for the development board you have. 

If that’s your first time with Particle, you can familiarize yourself with the getting started material first. 

Once ready, setup your Muon development board visiting part.cl/setup-m404 and create a new project using the Particle Workbench in Visual Studio Code and set it up to target the M-Som (you can also use the Web IDE or just the CLI if you prefer, but the details below assume you are using Particle). 

Connecting and setting up the accelerometer 

This project uses an ADXL362 SPI accelerometer connected to the Muon board. Particle has a library available for this sensor (along with hundreds of other ones). 

Connect the sensor to the Muon referencing the documentation for the sensor board and the Muon datasheet’s 40-pin expansion connector pinout diagram: 

Sensor  Muon 40-pin expansion connector  Description   
VIN  PIN 17 – 3V3  3.3V DC Power 
GND  PIN 25 – GND  Ground 
SCL / SCLK  PIN 23 – SCK  SPI CLK 
SDA / MOSI / SDI  PIN 19 – MOSI  SPI MOSIs 
SDO / MISO  PIN 21 – MISO  SPI MISO 
CS  PIN 26 – A2  SPI Chip select 
INT2  n/c  Interrupt 2 (not used) 
INT1  n/c  Interrupt 1 (not used) 

 

Find and install the ADXL362DMA library: in your project in VS Code, bring up the command palette (CTRL+SHIFT+P) and type Particle: Find Libraries, enter ADXL and check the search results in the terminal. You should see different options. In this project, you need the ADXL362DMA one. Bring up the command palette again and type Particle: Install Library and when prompted for it, enter ADXL362DMA. The library is downloaded and added to your project. 

Test the accelerometer: copy code from the 1-simple-ADXL362DMA.cpp file and paste it in your project’s main cpp file replacing all the existing content in the file. Compile and deploy. Open and connect the Serial Monitor and confirm you see the traces from the sensor. 

 

 

Create an Edge Impulse project 

Visit EdgeImpulse.com, login (or create an account if you don’t yet have one) 

Click on Create a new project, enter a name and click on the green button 

If you have not done so yet, you can check out the Motion: Gesture recognition tutorial, which this project was created from. 

 

 

Setup device for sensor data collection 

Before you can start working on your Edge Impulse project, you need to collect data from the sensor. Edge Impulse published a sample library for collecting data from an ADXL accelerometer (compatible with different iterations of the sensor) and offers a tool that can capture data on a serial connection on your development machine to forward it to Edge Impulse for ingestion. When using supported hardware, you can even connect your device directly to the Edge Impulse project over serial connection. 

  • In Visual Studio Code, bring up the command palette and type Particle: Install library, then enter the library name adxl-345-edge-impulse 
  • Copy the code from the file in lib/adxl345-edge-impulse/examples/data-forwarder/data-forwarder.cpp to your main cpp file in your project (replacing all existing content) 
  • Uncomment the following line to adapt the sample code to the ADXL362 sensor that we are using: 
#define SAMPLE_ACCELEROMETER

 

 

  • Compile and deploy to the board. In the serial terminal you should now see acceleration data flowing. 

Tip: if you see the message “no status yet, waiting for accelerometer” and no sensor data, that means the application was not able initialize the sensor correctly. That can be due to the sensor being stuck in MeasureMode and not being reset when rebooting the board. To resolve the issue, disconnect the battery from the device and disconnect the USB cable to power the board off completely. 

  • Make sure you stop the Monitoring in the Serial Terminal as you will need the COM port to be available for the next step. 

Data collection for training 

To train the model, you need datasets for each of the gestures you want the model to recognize. The more data you get, the better the model will be. Also, the data quality is key, and you won’t get better data than the data coming directly from the physical sensor on the device itself rather than static data or data from another device or sensor. 

To train models, Edge Impulse allows you to generate data from your phone (to collect images, audio or motion), to connect one of the many supported devices (using  a pre-built firmware), or to use a CLI tool called Data Forwarder that will capture data on the serial connection and forward them to your Edge Impulse project. We are using this tool in this project. 

  • In your Edge Impulse project, click on “Collect New Data” 

  • On the pop-up screen, you can see the 3 options mentioned above. We will use the third one which, if you click on it, it will show more information about the Data Forwarder tool. Discard the pop up.  

  • Click on the chip icon next to your profile picture to configure the target device for your Edge Impulse project. In Target device, select Particle Muon(if not available select Photon 2). 

  • Make sure you have the Edge Impulse CLI tool installed on your development machine. 
  • In a terminal, run edge-impulse-data-forwarder and enter your Edge Impulse account credentials. The tool will scan the available serial ports available, and you can pick the one that is your device. 

Tip: if you don’t know which COM port is your device and you see several in the list, you can check in the Visual Studio Code Serial Terminal what COM port number your device is showing as when connected. It should be the USB Serial Device that appears when you connect the device 

  • Select the Edge Impulse project you want to connect the device to. Then the tool will detect the type of data, will establish the connection with your edge impulse project, will ask you to name the 3 axes detected (for example: accelx, accely, accelz), and finally will ask you to name the device. Once you have answered all the questions, the Data Forwarder establishes the connection with your Edge impulse project and starts forwarding data. 

 

  • Back to your Edge Impulse project, you should now see the newly connected device in the Collect data Device drop down list and you can start sampling data. Type a label name (for example “up”) and click on Start sampling. As soon as the sampling is started, repeat the gesture you want to be recognized for “up” several times for 10 seconds. 

  • Repeat step 3 at least 10 times for each gesture: up, down and idle. 

Impulse creation and training in Edge Impulse studio 

At this point you are done capturing real data from your sensor and are ready to configure and train your model. The steps below are similar to the ones of the Edge Impulse continuous motion recognition end-to-end tutorial, so if you need more details or explanations, you can check out this tutorial. For this model, you will need a “Spectral Analysis” signal processing block that will apply filters, perform spectral analysis on the signal and extract frequency and spectral power data, and a “Neural Network” learning block that takes the spectral features and learns to distinguish between the 3 classes (up, down and idle). 

In Edge Impulse Studio, click on the Create impulse menu and you should see your time series data showing on the left. 

The Window Size and Window increase values can be adapted to the way you captured the samples. The window size should be the approximate time that each gesture takes, so if you repeated the gesture 10 times during the 10s sampling, the window time should be 1,000ms. The window increase determines the step or offset between consecutive windows. For instance, if your window size is 1000ms and your window increase is 500ms, the next window will start 500ms after the previous one. This means there will be some overlap between windows, which can help capture more continuous or overlapping events. 

Add a Spectral Analysis processing block and a Classification learning block then click on Save Impulse. 

Configure the spectral analysis block: click on the Spectral features menu on the left. Set the filter type to low and the cut-off frequency to 1Hz. These values vary depending on your sampling. You can learn more about the spectral  analysis processing block here and adjust the settings to your data. Once you are done, click on Save parameters. 

This should have sent you to the “Generate features” tab of the spectral analysis block. Click on Generate Features. This will split all raw data in windows based your settings, apply the spectral features block on all the windows, and calculate feature importance. Once the features are generated, the “Feature explorer” will load and you will be able to evaluate if your dataset is of good quality. If you can visually identify some clusters by classes, then the machine learning should be able to do so as well. If clusters are not clearly distinct or overlapping too much, you will want to capture more data and fine tune your spectral analysis settings. If you need to capture more data, you can go back and repeat steps 7 and 8 of the previous section. 

Configure the neural network: click on Classifier in the left menu, set the number of training cycles. You can learn more about the Classification learning block here. If your results are not good enough, you can increase the number of training cycles to optimize your model performance, and you might end up with 100% accuracy after training for 100 training cycles. This is not necessarily a good thing, as it might be a sign that the neural network is too tuned for the specific test set and might perform poorly on new data (that’s called overfitting). The best way to reduce this is by adding more data or reducing the learning rate. 

Test your model before exporting it to the device: Edge Impulse studio allows you to test your model right from the browser using the same Data Forwarder that allowed you to capture data for training. You can do so on the Live classification page (you can find it in the left menu). If you are not satisfied with the recognition of your gestures, you can go back and add more data, adjust your filtering and spectral analysis settings, and/or run more training cycles. 

Now that the model is trained, you can export the Edge Impulse library in the right format for our hardware: select the Deployment menu on the left and search for Particle library as a deployment option. You can choose between the EON compiler or TensorFlow Lite and pick if you want your model quantized or unoptimized. Once you have configured the export, click on Build. This will compile the model into a C++ library and download it with samples in a zip file. 

 

 

Integrate the Impulse in the Particle project 

The Edge Impulse model comes as a C++ library compressed in a Zip file. The archive contains the Edge Impulse SDK, the model itself and its parameters and an example application. 

Copy Edge Impulse SDK and model to your project: in the ReadMe instructions you will find instructions on how to load this very project in VS Code to build it and run it on your device, but what you really want is to import the model (and the Edge Impulse SDK) in your existing project. For this, you need to copy the following from the src folder and paste them in the folder of your project: edge-impulse-sdk folder, model-parameters folder, tflite-model folder, build.mk file. 

Add include statements and adapt your project’s code for tflite and Edge Impulse SDK: the Edge Impulse SDK and TensorFlow Lite code make their own definitions for some constants and methods already declared in the Particle namespace and you will need to make a couple changes to compile your application successfully. Here a couple workarounds to apply to your project’s main cpp file: 

The tflite library redefines the constantA0, A1 and A2 which conflict with Particle constants used for the Pin values of the Muon board connector. You need to undefine these just after including the particle.h header and before adding the new headers includes, then replace A2 with D17 when defining the accel object: 

#include "Particle.h" 

#include "ADXL362DMA.h" 

#include "adxl345.h" 

 

#ifdef A0 

#undef A0 

#endif // A0 

#ifdef A1 

#undef A1 

#endif // A1 

#ifdef A2 

#undef A2 

#endif // A2 

 

#include <stdarg.h> 

#include "edge-impulse-sdk/classifier/ei_run_classifier.h" 

#include "edge-impulse-sdk/dsp/numpy.hpp" 

#include "model-parameters/model_metadata.h" 

#include "edge-impulse-sdk/classifier/ei_classifier_smooth.h" 

 

… 

 

ADXL362DMA accel(SPI, D17);

 

 

The Edge Impulse SDK redefines the method Log which is used in Particle code to log information and errors to the debug connection (Log.info, Log.error()). Search for all the calls to the Log object functions in your code and add the particle:: namespace in front: 

Log.info("Hello, Muon!");      =>     particle::Log.info("Hello, Muon!");

 

 

Add code to use the Edge Impulse model in your code 

The below instructions assume you are using the code from the data forwarder sample you have used to push sensor data to the Edge Impulse project previously. 

Add the following include statements to reference the Edge Impulse model: 

#include <stdarg.h> 

#include "edge-impulse-sdk/classifier/ei_run_classifier.h" 

#include "edge-impulse-sdk/dsp/numpy.hpp" 

#include "model-parameters/model_metadata.h" 

#include "edge-impulse-sdk/classifier/ei_classifier_smooth.h"

 

 

Delete the definition of the ei_printf function and add an extern in front of its declaration: 

extern void ei_printf(const char *format, ...);

 

 

Before the Setup function, add the following declarations: 

static float features[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE]; 

char* runInference() 

{ 

    bool gestureRecognized = false; 

 

    ei_printf("Running inference.\n"); 

 

    // Check features array size 

    if (sizeof(features) / sizeof(features[0]) != EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE) { 

        ei_printf("The size of your 'features' array is not correct. Expected %lu items, but had %lu\n", 

            EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, sizeof(features) / sizeof(float)); 

        delay(1000); 

        return NULL; 

    } 

 

    ei_impulse_result_t result = {0}; 

 

    // the features are stored into flash, and we don't want to load everything into RAM 

    signal_t features_signal;     

    features_signal.total_length = sizeof(features) / sizeof(features[0]); 

    features_signal.get_data = [](size_t offset, size_t length, float *out_ptr) -> int { memcpy(out_ptr, features + offset, length * sizeof(float)); return 0; }; 

 

    // invoke the impulse 

    EI_IMPULSE_ERROR res = run_classifier(&features_signal, &result, false); 

    if (res != EI_IMPULSE_OK) { 

        ei_printf("ERR: Failed to run classifier (%d)\n", res); 

    } else 

    { 

        for (size_t i = 0; i < EI_CLASSIFIER_LABEL_COUNT; i++) 

        { 

            // We will only consider gestures with a confidence level > 90% 

            if (result.classification[i].value > 0.9) 

            { 

                ei_printf("Detected %s with %.5f accuracy\n", result.classification[i].label, result.classification[i].value); 

                gestureRecognized = true; 

                // If the gesture recognize is not idle, return the gesture 

                if (result.classification[i].label != "idle") 

                { 

                    return (char*)result.classification[i].label; 

                } 

            } 

        } 

        if (gestureRecognized == false) 

        { 

            ei_printf("Gesture not recognized.\n"); 

        } 

    } 

    return NULL; 

}

 

 

Replace the loop function with the following: 

void loop() { 

    ei_printf("Collecting accelerometer data for inference.\n"); 

    // Collect accelerometer data for Edge Impulse 

    for (size_t i=0 ; i<EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE; i+=3) 

    { 

        int16_t x, y, z; 

        accel.readXYZ(x, y, z); 

        // Convert Gs to M/s^2 and save to features array 

        features[i] = (((float)x * 2)/2048.0) * CONVERT_G_TO_MS2; 

        features[i+1] = (((float)y * 2)/2048.0) * CONVERT_G_TO_MS2; 

        features[i+2] = (((float)z * 2)/2048.0) * CONVERT_G_TO_MS2; 

 

        // Wait for 16ms (62.5Hz) before collecting the next sample 

        delay(16); 

    } 

    // Run inference 

    char* inferenceResult = runInference(); 

    if (inferenceResult != NULL) 

    { 

        Particle.publish(((strcmp(inferenceResult, "up")==0)?"gestureup":"gesturedown"), inferenceResult); 

    } 

}

 

 

Compile and deploy to your device. 

Tip: If you are using Windows and compiling locally, you might get a “parameter list too long” error from the arm gcc compiler while compiling. To successfully compile your project, you can use the Particle docker buildpack compile method: 

  • Ensure you have docker installed on your Windows machine, 
  • Download the container image with the following command (adapting version and target if not using an M-Som module): 
docker pull particle/buildpack-particle-firmware:5.9.0-msom

 

  • Run the following command from the project folder (you can use the VS Code terminal for that). This command will compile the application and build firmware binary file that we will then flash to the device: 
docker run --rm -v .:/input -v .:/output -e PLATFORM_ID=35 "particle/buildpack-particle-firmware:5.9.0-msom"

 

  • Using the Particle CLI tool, retrieve the connected device ID (the msom device should be connected to your development machine with a USB cable): 
particle usb list 

 

  • Take note of the device id for your M-Som 
  • Flash the firmware binary with the following command (replacing [DeviceID] with the ID retrieved on previous step: 

particle flash –local [DeviceID] firmware.bin 

 

Check the inference is working using the Visual Studio code serial monitor: 

At this point you can clean up the code to remove unnecessary traces and keep only the detection logs. 

Connecting the Particle Cloud to a Home Automation system  

In the application, each time a gesture up or down is recognized, a message is sent to the Particle Cloud with the Particle.Publish command: 

    // Run inference 

    char* inferenceResult = runInference(); 

    if (inferenceResult != NULL) 

    { 

        Particle.publish(((strcmp(inferenceResult, "up")==0)?"gestureup":"gesturedown"), inferenceResult); 

    }

 

 

Particle dramatically simplifies the integration with third party Cloud solutions. The most generic integration is the Custom Webhook which this project uses. 

At this point, as your own home automation system might differ, you should adapt the steps to your own. If you are using Amazon Alexa to control your home automation system, you can leverage an Alexa routine called the Webhook Routine Trigger created by Michael Dworkin and that he created in the context of a simple button open source project. 

This project is leveraging this service: The Particle Cloud will trigger the webhook which will trigger an Alexa routine of your choice. Here are the steps to establish the connection:  

Visit Webhook Routine Trigger Alexa Skill Launch routines with a URL and follow the steps to install the Webhook Routine Trigger skill in the Alexa App, discover devices in the Alexa app to see a new device of type doorbell appear (that’s the type of device the skill emulates), then  log in on the site using Amazon single sign on. Rename the webhook that was automatically created to TriggerShadeUp 

On the site you will see a webhook with a name and URL. This URL will be used to trigger a first routine that will open the shades. Create a second webhook called TriggerShadeDown 

In the Alexa app, tell it to “Discover my devices” and in the devices page you should see a couple new devices with the names of your webhooks you created previously: 

In the Alexa app, create a routine that will activate a device when the device TriggerShadeUp is “pressed”, and configure it to operate another device in your house (here, opening the guest room shades): 

Create a second Alexa routine activated when the device TriggerShadeDown is “pressed” and configure it to control the device of your choice (here, closing the guest room shades). 

In the Particle console, go to the integrations page, click on Add new integration, select Custom Webhook and configure it to be triggered when your device sends a gestureup event. For the URL, use the one from the skill page for the TriggerShadeUp Webhook. 

Add a second Webhook integration that will be triggered on a gesturedown event from your device and that will POST on the URL for the TriggerShadeDown Webhook. 

Et voila! When you do the up gesture, your shades will open and when you do the down gesture they will close! 

Edge Impulse and Particle in action 

Let’s look back at the main challenges Edge AI development presents, and how Edge Impulse and Particle made them all simpler: 

✅ Collecting real data from sensors: Using the Particle Workbench, and thanks to the Particle platform it was straight forward to build, compile and deploy embedded code getting sensor data and sending it to the serial port, while the Edge Impulse Data Forwarder and Project made it transparent to forward all this sensor data to the Cloud from the development machine 

✅ Optimizing Machine Learning Models for constrained devices: that’s what Edge Impulse is all about and the deployment of the optimized model as a Particle library made its integration on a Particle device seamless. 

✅ Compiling ML Models for Embedded Applications: while Edge Impulse takes care of optimizing the model and exporting it in the right format for the target platform, the Particle workbench as well as the other tools such as the Docker BuildPacks make the compilation process transparent. 

✅ The iterative process of tuning ML models on constrained devices: you have probably seen how simple it was, during the training of your model in the Edge Impulse Studio to go back to capture and add more data, refine the settings for the features time windows, the filters and analytics or the training cycles and iterate on the process. You have also seen how comfortable compiling, flashing, and iterating on the Particle firmware is. 

 

This experiment demonstrates the potential of Edge Impulse and Particle to make Edge AI accessible and feasible for microcontroller-powered devices. The combination of these technologies enables the development of intelligent and responsive IoT solutions, showcasing their promise for future applications in industrial and enterprise environments. 

 

Comments are not currently available for this post.