A Distributed Edge-IoT Paradigm for Multimodal Sign Perception and Live Speech Reconstruction

Abstract

The Internet of Things (IoT)-based intelligent glove combined with an Android application is a wearable assistive technology developed to convert hand movements into instant text and voice outputs, supporting communication for people with hearing and speech disabilities. The glove incorporates several flex sensors mounted along the fingers to capture bending variations corresponding to sign language gestures, including alphabets, common words, and basic phrases. These analog signals are continuously collected and interpreted by an ESP32 microcontroller, which acts as the central processing unit for decoding gesture patterns. A rule-driven algorithm is utilized to evaluate the sensor data and match it with predefined gesture mappings, generating meaningful textual information. The resulting text is instantly shown on a Liquid Crystal Display (LCD) for visual confirmation and is also transformed into audible speech using a speaker module, enabling verbal communication. In addition, the system integrates IoT functionality to send the processed outputs to a connected Android application, allowing remote viewing, data access, and improved user interaction. A reliable power supply module ensures consistent and uninterrupted functioning of all hardware components. The design emphasizes lightweight construction, affordability, and the capability to operate independently without continuous internet access, making it adaptable for various environments such as residential spaces, educational institutions, healthcare facilities, and public settings. This solution enhances inclusive interaction by allowing users to express their thoughts and requirements effectively, promoting independence and improving quality of life.