Edge-Enabled IoT Multimodal Wearable System for Real-Time Sign Language to Speech Conversion with Mobile Integration

Abstract

This research is a wearable assistive system designed to translate hand gestures into real-time text and speech, enabling effective communication for individuals with hearing and speech impairments. The glove is embedded with multiple flex sensors placed along the fingers to detect variations in bending patterns associated with sign language gestures, including letters, words, and simple expressions. These sensor readings are continuously acquired and processed by an ESP32 microcontroller, which serves as the core unit for signal interpretation. A rule-based approach is employed to analyze the incoming data and map it to predefined gesture patterns, which are then converted into meaningful textual outputs. The interpreted text is immediately displayed on an LCD for visual feedback and simultaneously converted into audio through a speaker module to facilitate verbal interaction. Furthermore, the system incorporates IoT capabilities to transmit the generated outputs to a connected Android application, allowing remote access, monitoring, and enhanced usability. A stable power supply unit supports uninterrupted operation of the system components. The design prioritizes portability, cost-effectiveness, and the ability to function independently without requiring constant internet connectivity, making it suitable for diverse environments such as homes, schools, hospitals, and public areas. Overall, the system promotes accessible and inclusive communication by enabling users to convey their needs efficiently and independently.