Multimodal Attention-Based Mixture of Experts for Video-Level Activity Recognition using the YouTube-8M Dataset

Abstract

 Human Activity Recognition (HAR) using large scale video data has become an important research topic in computer vision and video data analysis. The rapid growth of online video content has created a need for effective methods that automatically analyze and recognize activities from large-scale video datasets. So in this paper, a multimodal deep learning framework is proposed video-level human activity recognition using youtube-8M dataset. The proposed method utilizes both video and audio features to learn complementary information from acoustic and visual modalities. First, Input features are projected into a shared feature space and divided into tokens for attention-based learning. Then self-attention is applied to capture relationships within each modality, while cross-attention enables interaction between video and audio representations. The features are then combined using a gated fusion mechanism to generate a unified multimodal representation. Lastly, a Mixture-Of-Experts architecture is used to aggregate predictions from multiple expert networks for activity classification. Result shows that the proposed model achieves Validation GAP 82.89% while using 25 Million parameters which is lower compared to other existing methods, showcasing its effectiveness for large-scale video activity recognition.