CLIP

CLIP (Contrastive Language-Image Pretraining) is a dual-encoder model from OpenAI (Radford et al. 2021) that learns a joint embedding space for images and text. Two encoders — a vision transformer for images, a text transformer for text — are trained with a contrastive objective on 400 million image-caption pairs scraped from the web: matching image-text pairs are pulled close in the embedding space, mismatched pairs are pushed apart. The result is a shared representation where a photograph of a golden retriever and the text 'a golden retriever' end up close, regardless of the photograph's specifics. The model enabled a wave of downstream applications — zero-shot classification, image search by text query, text-to-image model conditioning, multimodal RAG — without any task-specific fine-tuning.

The 2026 landscape has open-weight CLIP variants significantly beyond the original. **OpenCLIP** (LAION) reproduced and scaled CLIP on LAION-2B and LAION-5B public datasets, producing models that often outperform the original OpenAI CLIP. **SigLIP** (Google) replaces CLIP's softmax contrastive loss with pairwise sigmoid losses and trains more efficiently with better benchmark results — widely adopted for 2024+ production vision systems. **EVA-CLIP** (BAAI) scales to multi-billion parameters with strong robustness. **BGE-VL**, **Jina-CLIP v2**, **Nomic Embed Vision** extend CLIP-style embedders to longer contexts and more languages — material for APAC where English-centric CLIP variants underperform. **CN-CLIP** specifically targets Chinese image-text.

For APAC mid-market teams, CLIP-family embedders are the **default choice for image search, visual RAG, and text-to-image model conditioning**. SigLIP has largely replaced OpenCLIP as the preferred starting point in 2025-26 for better quality and training efficiency. Multilingual variants matter: generic English-trained CLIP underperforms on Japanese / Korean / Traditional Chinese captions and visual domains; use explicitly multilingual variants (SigLIP multilingual, Jina-CLIP v2, CN-CLIP for Chinese) for APAC workloads. Fine-tuning on your own image-caption pairs is worthwhile for domain-specific visual vocabulary (retail product catalogues, industrial equipment) where generic pretraining underperforms.

The non-obvious failure mode is **CLIP embeddings miss fine-grained visual details**. Counting (how many cars in the image), spatial relations (left of / above / inside), reading text within the image, and subtle attribute discrimination are all weaknesses of the CLIP-style contrastive representation. Applications that need these — document understanding, chart extraction, complex-scene reasoning — should use a vision-language model (VLM — GPT-4o, Claude 3.7, Gemini 1.5, Qwen-VL, InternVL) rather than CLIP embeddings. Choose CLIP for semantic retrieval; choose a VLM for interpretive understanding. Using CLIP where a VLM is needed produces embeddings that look similar but encode none of the distinguishing detail.