Cloud-Based Architecture for Italian Sign Language Translation with Fine-Tuned LLMs for LIS-to-Italian Conversion

This thesis aims to develop a client-server system for translating Italian Sign Language (LIS) into Italian, focusing on improving communication accessibility for individuals who are deaf or hard of hearing. The work was conducted as part of the research project LIS2S, led by Orbyta, which aims to advance sign language translation technologies. The system integrates machine learning, computer vision, and web technologies to bridge the communication gap between LIS users and the broader Italian-speaking community. The proposed system captures video input, which is processed to extract motion data. This data is then used to recognize signs, which are converted into LIS glosses (written representations or transcriptions of the signs in sign language) that capture the meaning of the gestures made. These glosses are subsequently translated into written or spoken Italian. The system utilizes Django for web application management, Docker for containerization, and WebSocket for efficient data transmission. Large language models (LLMs) and generative models will be explored to generate synthetic data, supporting the training of models and addressing the challenges in the translation process. A key challenge addressed in this work is the lack of extensive annotated datasets for training models focused on LIS-to-Italian translation. Additionally, translating LIS glosses into fluent and grammatically correct Italian sentences presents significant complexities. By leveraging machine learning techniques, the system has been trained to map these glosses to the corresponding Italian language structure. The system's development provides a functional platform for translating LIS to Italian in near real-time, offering a valuable tool for improving communication accessibility. This work contributes to the field of sign language translation by creating an end-to-end solution that enhances the interaction between the deaf and hard-of-hearing community and the Italian-speaking population, furthering inclusivity and bridging language gaps.