Ninth International Conference SmartCom 2025

Authors - Aye Thiri Nyunt, Nishi Vora, Devanshi Vaghela, Brij Kotak, Ravi Chauhan, Kirtirajsinh Zala
Abstract - This paper is an AI and Machine Learning Algorithm - based dualistic Gesture-to-Speech and Speech-to-Gesture framework. The core of this initiative is to enable machines and humans to converse with each other by enabling the translation of physical body movements into reasonable speech and vice versa. We used deep learning models- Convolutional Neural Networks (CNN)- to train our system using a dataset consisting of human gestural movements and the relevant speech patterns. For the Gesture-to-Speech module, real-time gesture recognition and interpretation were used, which involved computer vision and were implemented to interpret gestures into speech output containing words and phrases representing the message illustrated by the gestures. The Speech-to-Gesture module, on the other hand, uses speech as input to produce context-related gestures-the main purpose of which is to improve user interaction and experiences. In the system, multiple applications were tested, including sign language and webcams. Further research will try to extend the flexibility of the system to include various languages, cultural backgrounds and characteristics of individual gesture styles which eventually has a high level of customization. We had designed the CNN architecture for real-time gesture recognition and taken care of data preprocessing as well to increase accuracy concerning different types of gestures. We created Gesture-to-Speech translation with the use of an LSTM, then added in a Text-to-Speech engine for it to have a very natural sound. We then worked on Speech-to-Gesture and even refined the gestures through a CNN-based network, to ensure transitions are very fluid. Everything was coordinated such that there would be synchronous gestures and speech for extremely natural real-time interaction. We coached on how one would integrate, test, and further optimize models with dropout and batch normalization for higher performance.