Automatic Unmuting: Unlocking the Power of Edge Machine Learning
Featured in Make Magazine 2023, pioneering the future of IoT and Edge Computing.
Automatic unmute tool using Edge ML
Have you ever encountered the awkwardness of realizing you've been speaking while on mute during an online meeting or conference? We present to you an innovative solution that employs machine learning to detect the phrase "You're Muted" and subsequently sends an automated command to your device to unmute you.
This groundbreaking project is powered by Particle's newly introduced Photon 2, an advanced WiFi development board equipped with a robust processor capable of handling Edge ML operations. This innovation underscores Particle's collaboration with Edge Impulse, aiming to revolutionize detection and recognition capabilities from Cloud to Edge and vice versa.
Follow this comprehensive guide to set up and customize this tool for your specific needs.
About Particle and Edge ML
Particle is an innovative hardware and software company focused on enabling the Internet of Things (IoT). Our easy-to-use development tools empower makers, developers and engineers to quickly prototype and productize connected devices.
A part of the Particle platform for our latest products is support for Edge ML, allowing devices to run machine learning models locally for ultra-low latency inference.
This powerful combination of hardware, Device OS, developer tools and Edge ML enables the creation of intelligent IoT endpoints. Particle is pioneering the future of IoT and edge computing - simplifying and accelerating the building of innovative connected products at scale.
Tutorial:
This tutorial will provide a step-by-step guide, from setting up the hardware components to configuring the software and testing the device with your conferencing tools.
Skip to step 5 to read about how to modify this example for other use cases!
1. Setting Up the Hardware
Position your Photon 2 and Digital Microphone module on a breadboard as instructed, and interconnect them using jumper cables. The Photon 2's specifications include a 200MHz CPU and 4.5MB of SRAM, ensuring smooth processing of the machine learning task of recognizing audio patterns.
For this setup, various Digital Microphones compatible with PDM (Pulse Density Modulation) can be employed. We recommend the Adafruit PDM microphone due to its accessibility and superior noise floor, ensuring optimal voice pickup even from a distance.
Pinout configuration:
| Mic Module | Wire Color | Photon 2 | Function |
|---|---|---|---|
| 3V | Red | 3V3 | 3.3V Power |
| GND | Black | GND | Ground |
| SEL | - | - | Left/right select (usually unconnected) |
| CLK | Yellow | A0 | PDM Clock |
| DAT | White | A1 | PDM Data |
2. Software Installation for Photon 2
Connect the Photon 2 module to your computer. Use the Chrome web browser and access the link above.
This portal will guide you through programming the Photon 2 using WebUSB technology built into the Chrome browser. Once programmed, you might be prompted to select a keyboard type, choose 'ANSI' if required.
Post programming, your Photon 2 will be pre-loaded with the “You’re muted” application, inclusive of the local ML processing engine. The application is compatible with platforms like Zoom, and further customizations will be discussed in the subsequent steps.
3. Configuring Communication Software
For optimal performance, configure your chosen communication software to respond to the Photon 2's commands.
For Zoom: Navigate to settings and locate "Mute/Unmute my Audio." Enable the feature and note the shortcut. This can be customized, but ensure the firmware is updated accordingly.
For platforms like Google Meet and Microsoft Teams: Default keyboard shortcuts are utilized. However, any modification requires the Photon 2 firmware to be recompiled.
4. Testing the System
Upon initiating the Photon 2, it displays LED signals:
- Green indicates active listening
- Orange signifies successful detection and keypress transmission
- Red indicates an error (verify your setup in this case)
Speak the phrase “You’re muted” to test the system. The underlying mechanism involves a TinyML model that processes audio samples from the microphone. The model assesses the confidence level of the detected phrase, and upon sufficient confidence, transmits a keypress command.
For a more tailored experience, refer to our GitHub repository, which offers guidance on adding personalized voice samples and modifying detection phrases.
5. Customization Options
To further adapt the tool to your needs, you can modify the firmware.
- Download the GitRepo onto your computer
git clone https://github.com/particle-iot/make-magazine-muted-demo.git - Follow the instructions here to install Visual Studio Code and the Particle Plugin
- Open the github project in Visual Studio code and compile it.
- Bring up the Command Palette and select “Particle: Configure Project for Device” - from the list, select the top option (device os 5.5.0 at the time of writing)
- Bring up the Command Palette again select “Particle: Flash application (local)”. This will compile the application first if its not built before flashing it onto the Photon 2 that you connect via USB.
- You can modify the code base to change the conference tool that is being controlled by changing the lines of code in src/make-magazine-muted-demo.cpp. Configure the keyboard shortcut in the function
generateKeystrokes()near the top of the file.
- In order to alter the code base substantially to recognize a new keyword, you can follow the tutorial over on Edge Impulse's website.
Conclusion:
With this introduction to Edge ML and the Particle device ecosystem, we hope to inspire and equip you to further automate and enhance your digital interactions. We envision countless applications, from smart home integrations to automated task assistants.
We wish you productive meetings, and may the Photon 2 always ensure you're heard!