Project
| # | Title | Team Members | TA | Documents | Sponsor |
|---|---|---|---|---|---|
| 57 | Wireless EMG and IMU Sleeve for Hand Gesture Recognition |
Diqing Zuo Harbin Li Jameson Koonce |
Michael Molter | proposal1.pdf |
|
| # Team Members: - Jameson Koonce (jrk8) - Diqing Zuo (diqingz2) - Harbin Li (hdli2) # Problem As advancements have been made in the Virtual Reality (VR) space, more practical applications of the technology have been found such as in education, engineering, utilities maintenance, and entertainment ([source](https://pmc.ncbi.nlm.nih.gov/articles/PMC9517547/#sec4-ijerph-19-11278)). However, this technology is not yet immersive enough as the majority of users experience some level of cybersickness during use characterized by discomfort ([source](https://pmc.ncbi.nlm.nih.gov/articles/PMC8886867/#Sec1)). Part of this immersion loss can be attributed to how VR consoles track the user’s hands, with some solutions involving controllers, leading to a lack of immersion, and others involving computer vision, which can be inaccurate in many hand/arm positions. There needs to be a more effective way to immerse a VR user’s arm and hands into a virtual environment. # Solution We are looking to create a system which tracks arm movements and recognizes hand gestures for more immersive Virtual Reality (VR) Environments. Specifically, we are going to develop a wireless sleeve lined with Electromyography (EMG) and Inertial Measurement Unit (IMU) sensors in order to detect electrical signals, orientation, and acceleration information from a user's arm and use on-device processing of machine learning algorithms to classify individual finger gestures and track arm movement. This system will be more immersive than existing solutions because the user’s hands will be free in a VR environment, and the arm motion will be tracked even when the arm is out of view. The system will make use of EMG and IMU sensors on a physical sleeve, connected to a wireless module to assure that the information can be used as a controller for external devices and the user is physically unconstrained. The data will be processed in our on-sleeve ML framework for classification and tracking, but raw data can be processed off-sleeve for higher computational efficacy, with an increase in latency. # Solution Components ## Sensor Array System Description: Array of sensors responsible for collecting and preprocessing the analog signals for use by the processing unit. - Dry sEMG Electrodes: large array of dry electrodes for recognition of movements in the hand. - IMUs (ICM-20948 9-axis IMU): collection of accelerometer, gyroscope, and magnetometer to track the orientation and movement of the arm. - Op-amp Denoising (OPA4277UA): operational amplifiers for signal conditioning. ## ML-Based Gesture Recognition (Software) Description: Processes EMG data collected using ML models to classify hand/finger/arm gestures in real time. Components: - Microcontroller : responsible for interfacing with the EMG sensors, preprocessing raw signals, and system control STM32WB55 Series MCU - ML Framework: Optimized for real-time, low power consumption inference. - TensorFlow Lite for Microcontrollers (tflite-micro/tensorflow/lite/micro/examples at main · tensorflow/tflite-micro · GitHub) Possible external dataset Ninapro (Ninapro) - Edge processing module (is only needed for high real-time inference latency requirements): executes the ML model directly on device for low-latency inference —nRF52840 SoC - Training model on EMG signal data We are training our model solely on our own collected EMG data from a single user, focusing on a limited number of gestures first to demonstrate feasibility. The ninapro dataset could serve as a reference dataset for understanding gesture patterns but would currently not be used directly in training. The training and optimization of the ML model would be divided into the following parts: 1. Data Collection: Data would be collected from a single user and focusing on a small subset of predefined gestures. This would then be labelled and used to train our model. 2. Feature Extraction: Extract relevant features from EMG signals, including amplitude, frequency domain characteristics, and time-domain patterns. 3. Model Architecture: Uses a lightweight deep learning model. We consider two primary approaches CNN and RNN, while our primary attempt would be focusing on CNN due to a lower requirement for processing power and memory. Based on the above, we train the ML model and then converter the trained model into TensorFlow Lite model. - Classification of EMG signals (text/command) We start by first preprocessing the raw EMG signals by applying filtering techniques to remove noise and enhancing signal quality. The extracted features such as signal amplitude, frequency, patterns are analyzed to identify the gesture characteristics. The processed data would then be fed into our trained ML model that classifies the EMG signals into specific gestures and thus converted into text-based commands, control signals, etc. ## Wireless module Description: Manages real-time communication between the wearable device and external systems, enabling efficient transmission of classified gesture data for further processing or user interaction. Components: - Wireless Protocol: We would be using BLE for efficient, low-power wireless communication - Integrated BLE MCU: The STM32WB55 includes a built-in BLE radio ## Physical component (wearable form) Description: Physical - Nylon-spandex sleeve with electrode cutouts - PCB and electronic mounts - Li-Po battery and attachment # Criterion For Success - Reliability/consistency in discerning gesture - Show viability by implementing it on one person only. - Achieve 95% accuracy in recognizing a set of 6 gestures - Demonstrate wearability for extended periods (1+ hours) without significant signal degradation (maintaining 90%+ accuracy). - Achieving same or similar accuracy between sessions of wearing, with minimal or to no calibration. - Wireless capability - Demonstrate wireless capability and clearly show gesture recognition and arm tracking results on external device - Latency - Achieving latency of below 200ms |
|||||