A Hybrid Cross-Modal Deep Learning and CLIP-Based Framework for Personalized Fashion Product Recommendations

Abstract

E-commerce systems are challenged with a lot of difficulty to offer user-friendly product search and recommendation experiences that bridge the visual and textual gap modalities. The paper introduces a new cross-modal recommendation system that uses CLIP (Contrastive Language-Image Pre-training) to conduct multi-modal search of products. Our system allows users to query based on natural language queries, image, or both queries together through the intelligent fusion systems. To improve relevance of recommendations, we include context-considered ranking using visual sentiment analysis, occasion-mood matching and time sensibility. The system utilizes efficient similarity search using FAISS with a variety of fusion options (weighted averaging, concatenation, element-wise multiplication) and high-tech reranking mechanisms (cross-attention). The experimental results show the better results of our approach; it gives a much better score of 92.3 percent precision at 10 on text queries, 88.7 percent on image queries, and 94.1 percent on single hybrid queries. Occasion-aware ranking enhances the user satisfaction by 34, and visual sentiment analysis brings out more engagement by 28. Our RESTful API platform can support real-time inference with an average query latency of 127ms, which is appropriate to be deployed at production in large-scale e-commerce systems.