Avatar-Driven Synthetic Data Augmentation for Enhanced Multilingual Sign Language Recognition

Abstract

Sign Languages serve as the primary means of communication for millions of deaf and hard-of-hearing individuals. Despite this significance, the development of automatic Sign Language Recognition systems is limited due to the scarcity of large-scale, high-quality video datasets. This is mainly attributed to the inherently resource-intensive process required to collect such datasets, often requiring fluent signers, controlled recording environments, and rigorous annotation. In response to these challenges, this study introduces a novel method for generating synthetic sign language datasets. Supporting both isolated and continuous signing, our method enables the creation of diverse and scalable datasets, with precise control over factors such as signer appearance, sign content, and environment. First, we determine the most suitable task for evaluation by comparing three architectures: CNN-Transformer, CNN-GRU, and CNN-LSTM. With the CNN-LSTM emerging as the most effective, achieving recognition rates of 75%, 90%, 100%, and 100% for Top-1, Top-2, Top-3, and Top-5 metrics, respectively. Our proposed method is then compared to standard data augmentation techniques demonstrating remarkable improvements, with recognition rate improvements ranging from 87.5% increase in Top-1, to a 42.85% improvement in Top-5. 10-fold cross-validation was then used to confirm the robustness of the resulting model in real-world scenarios.