Bridging the Communication Gap: A Mobile App for Seamless Integration of Sign Language in Real-Time Video Communication

Bridging the communication gap between deaf and hearing individuals remains a significant challenge, particularly within specific domains such as education, healthcare and public administration. Although sign languages are fully recognized as natural languages, their limited integration into mainstream communication tools contributes to the social and linguistic marginalization of deaf communities worldwide. Addressing this gap requires the development of inclusive, privacy-aware, and real-time technological solutions that facilitate seamless interaction between individuals, regardless of their different modalities of communication. This thesis aims to contribute to this objective by building an Android mobile application that integrates an AI-powered bidirectional translation engine. Specifically, the project involves the design and implementation of two distinct, interconnected modules. The first module is a backend system composed of multiple microservices, each running in a Docker container. The core service is a Node.js-based web server responsible for handling user authentication, video calls and user data, which is stored in a relational PostgreSQL database. In particular, for video calls, the server manages both the signaling process, using push notifications, and the routing and forwarding of media streams among participants. Additionally, to support real-time sign language translation, the server communicates with two Python-based microservices that process media streams by executing the corresponding AI translation algorithms. The second module is an Android mobile application built with Expo, which interacts with the backend via RESTful APIs. Users can use it to engage in video-based interactions similarly to popular video conferencing tools, but enhanced by real-time sign language translation features. The result of this work is a fully functional prototype designed to provide a solid foundation for future enhancements and further development.