Domain-Adaptive Zero-Shot Image Classification via CLIP and Large Language Models

Resumen

An adaptive prototype learning mechanism, which learns to improve class prototypes via exponential moving average updates, a domain-aware prompt designing technique utilizing LLM-produced contextual descriptions attuned to each distinct visual domain – nature, medical, satellite, and artwork domains – and culminates in a confidence-calibrated inference framework which beautifully integrates CLIP's vision semantic matching capabilities with LLM-driven reasoning capabilities for accurate inferences. We show a series of rigorous evaluation tests conducted on five distinct visual domains, demonstrating our model outperforms zero-shot classify accuracy relative to existing CLIP-existing baselines with negligible additional computation cost. Specifically, our model shows an average relative gain in top-1 accuracy performance of 4.2%, but with profoundly large gains of 6.8% in medical image and 5.3% in satellite imaging tasks. Further, experimental tests illustrate prototype learning methods can be successfully deployed for continuous learning with an increasingly large volume of unlabelled image inputs up to a gain in accuracy of 8.5% with 500 processed images. Taken in toto, our ablation studies confirm all constituent elements delineated in our proposed model have a vital part in a satisfactory accuracy performance level attainment.