Driver Inattention Detection Using a Hybrid Vision Transformer and BiLSTM

Resumo

Abstract: The identification of the early warning of unsafe behavior on the highways of visual cues of driver inattention is a serious challenge since spatial and temporal changes are not trivial many seconds before unsafe behavior. In this paper we propose a hybrid ViT-BiLSTM model that considers fixed-length sequences of 15 frames: inchoate frames are cropped according to the bounding-box annotations, safe or unsafe label and selectively augmented to have equal sampling. Every frame is processed using a Vision Transformer backbone (ViT B/16) and generates 768 dimensional features, which are further implemented as an input to the two-layer bidirectional LSTM ( hidden size of 256). The Dual classification heads produce both the frame level and sequence level prediction through sigmoid activation. The training is done in two steps, which are to freeze ViT and to train the BiLSTM and heads with AdamW (lr=1e 3), and then to fine-tune the last four ViT layers with different learning rates (ViT:5e 5, LSTM:1e 4) under mixed precision. Accuracy, precision, recall, and F1-score will be used at both levels with data visualized using confusion matrices, PR/ROC curves, and embedding projections (t-SNE/PCA). The last model ends with a mean of about 98 percent frame-level accuracy (F1 98) and 94 percent sequence-level accuracy (F1 94). The interpretability and soundness of the model are supported by qualitative visualizations of frame strips with annotations and trends of probabilities, which proves that the model may be used in real-life driver monitoring systems.