minor

Author: Yif-Yang

README.md

@@ -1,58 +1,60 @@
-# LLM2CLIP: Powerful Language Model Unlocking Richer Visual Representations
+# LLM2CLIP: Unlocking Richer Visual Representations with a Powerful Language Model
 
-Welcome to the official repository for **LLM2CLIP**! This project leverages large language models (LLMs) as powerful textual teachers for CLIP’s visual encoder, enabling more nuanced and comprehensive multimodal learning. 
+Welcome to the official repository for **LLM2CLIP**! This project leverages large language models (LLMs) as powerful textual teachers for CLIP's visual encoder, enabling more nuanced and comprehensive multimodal learning.
 
 [![Paper](https://img.shields.io/badge/Paper-arXiv-red)](https://arxiv.org/abs/2411.04997) [![Project Homepage](https://img.shields.io/badge/Project-Homepage-blue)](https://aka.ms/llm2clip) [![HuggingFace Collection](https://img.shields.io/badge/HuggingFace-Collection-orange)](https://huggingface.co/collections/microsoft/llm2clip-672323a266173cfa40b32d4c)  
-Paper: Accepted to NeurIPS 2024 Workshop SSL 
+**Paper:** Preprinted and under-review now. Accepted to NeurIPS 2024 Workshop: Self-Supervised Learning - Theory and Practice
 
 ---
 <img src="docs/static/images/radar_paper(4).png" style="max-width: 800px;">
 
 ## Challenges with Existing CLIP
 
 Current versions of CLIP face several limitations:
-- The text encoder has a short context window of only 77 tokens, limiting its ability to understand lengthy inputs.
-- The text encoder is relatively weak, often criticized for its inability to comprehend complex text, functioning nearly as a bag-of-words model.
+
+- **Limited Context Window**: The text encoder has a short context window of only 77 tokens, which restricts its understanding of lengthy inputs.
+- **Weak Text Comprehension**: The text encoder is relatively limited in its ability to comprehend complex text, often functioning as a bag-of-words model with limited depth.
 
 ## Why Integrate LLM with CLIP?
 
-Providing unimaginable cross-language capabilities. Our LLM2CLIP fine-tuned on purely English corpus even outperforms Chinese CLIP.
+LLM2CLIP brings the unimaginable power of large language models to CLIP, even surpassing native language capabilities. Our LLM2CLIP, fine-tuned purely on an English corpus, outperforms standard Chinese CLIP models:
 
-1. **Extended Input Window**: The LLM greatly expands CLIP's input window, allowing richer textual context.
-2. **Enhanced Understanding**: With LLM's help, CLIP can better comprehend dense and complex captions, improving text-image alignment.
-3. **Open-World Knowledge**: LLM supplements open-world knowledge, allowing CLIP to align multimodal features more globally, enhancing training efficiency.
+1. **Extended Input Window**: The LLM expands CLIP's input window, allowing richer textual context and better comprehension of long inputs.
+2. **Enhanced Understanding**: With LLM's help, CLIP gains a deeper understanding of dense, complex captions, leading to improved text-image alignment.
+3. **Open-World Knowledge**: The LLM provides open-world knowledge, enabling more globally informed multimodal feature alignment and boosting training efficiency.
 
 ## Key Challenges
 
-LLMs have strong text encoding capabilities hidden within the model, but their output space is often not highly separable for contrastive learning.
+While LLMs have strong inherent text encoding capabilities, the output space is often not highly separable, which limits their effectiveness for contrastive learning.
+![coco_score.svg](docs%2Fstatic%2Fimages%2Fcoco_score.svg)
 
 ## Our Approach
 
-We designed a Caption-to-Caption contrastive learning strategy, training the LLM to better differentiate between captions of the same or different images. This This caption-caption discrimination enhances the output space's separability enhances the output space's separability. 
-The LLM gradients were frozen while efficiently training CLIP's visual encoder on limited data, resulting in substantial performance improvements.
+To overcome these challenges, we designed a **Caption-to-Caption Contrastive Learning** strategy. We trained the LLM to better differentiate between captions of the same or different images, enhancing the separability of the LLM's output space. During training, the LLM gradients were frozen while CLIP's visual encoder was fine-tuned on limited data, resulting in significant performance gains. 
+
+Through this strategy, we better utilized the LLM's power to comprehend and process **long and dense captions**, improving the overall representation capabilities.
 
 ## What Can You Achieve with LLM2CLIP?
 
-1. **Enhanced CLIP Models**: Use our code to fine-tune pretrained CLIP models with representative dense captions or task-specific image-text datasets, making CLIP stronger for various tasks.
-2. **Out-of-the-Box Power**: Directly use our enhanced CLIP models, which have been made significantly more powerful with LLM guidance.
+1. **Enhanced CLIP Models**: Fine-tune pretrained CLIP models with dense captions or task-specific image-text datasets, making CLIP stronger for various use cases.
+2. **Out-of-the-Box Power**: Directly use our enhanced CLIP models, significantly upgraded with LLM guidance for superior performance in multimodal tasks.
 
 ---
 
 ## News 🚀🚀🚀
-- **[2024-11-06]** OpenAI's CLIP and EVA02's ViT base and large models are now available on  [HuggingFace](https://huggingface.co/collections/microsoft/llm2clip-672323a266173cfa40b32d4c). More model versions and datasets will be added to HuggingFace shortly.
-- **[2024-11-01]** Our paper has been accepted to the NeurIPS 2024 SSL Workshop!
+- **[2024-11-06]** OpenAI's CLIP and EVA02's ViT base and large models are now available on [HuggingFace](https://huggingface.co/collections/microsoft/llm2clip-672323a266173cfa40b32d4c). More model versions and datasets coming soon!
+- **[2024-11-01]** Our paper was accepted to the NeurIPS 2024 SSL Workshop!
 
 ---
-![main.svg](docs%2Fstatic%2Fimages%2Fmain.svg)
-
-## Model Zoo (Keep Updating)
+![main.svg](docs/static/images/main.svg)
 
+## Model Zoo (Continuously Updated)
 
 Stay tuned for updates on pretrained models and datasets, which will be made available in the [HuggingFace Model Zoo](https://huggingface.co/collections/microsoft/llm2clip-672323a266173cfa40b32d4c).
 
 ---
 
-## 💻 How to Install
+## 💻 Installation Guide
 
 1. **Create the environment**:
 
@@ -63,17 +65,16 @@ Stay tuned for updates on pretrained models and datasets, which will be made ava
    ```
 2. **Data Preparation**:
 
-### Data Preparation (Coming Soon) 
+   *(Coming Soon)*
+
+3. **🔥 Training**:
 
-### 🔥 Training
-   
    ```bash
    sh run.sh
    ```
+## ❤️ Acknowledgements
 
-## ❤️ Acknowledgement
-
-Currently, our code is built on top of [eva-clip](https://github.com/baaivision/EVA/tree/master/EVA-CLIP) and .
+Our code is built on top of [EVA-CLIP](https://github.com/baaivision/EVA/tree/master/EVA-CLIP). We would like to thank the EVA team for their foundational work.
 
 ## Citation
 
@@ -89,4 +90,3 @@ If you use our work, please cite:
       primaryClass={cs.CV},
       url={https://arxiv.org/abs/2411.04997}, 
 }
-```

docs/index.html

@@ -283,7 +283,8 @@ <h2 class="title is-3"><span class="dvima">Why Integrate LLM with CLIP?</span></
 
                   <h2 class="title is-3"><span class="dvima">Our Approach</span></h2>
                   <p style="font-size: 100%">
-                    We designed a Caption-to-Caption contrastive learning strategy, training the LLM to better differentiate between captions of the same or different images. This caption-caption discrimination enhances the output space's separability. The LLM gradients were frozen while efficiently training CLIP's visual encoder on limited data, resulting in substantial performance improvements.
+                    We designed a Caption-to-Caption contrastive learning strategy, training the LLM to better differentiate between captions of the same or different images. This caption-caption discrimination enhances LLM output space's separability. The LLM gradients were frozen while efficiently training CLIP's visual encoder on limited data, resulting in substantial performance improvements. Through this strategy, we better utilized the LLM's power to comprehend and process **long and dense captions**, improving the overall representation capabilities.
+
                   </p>
 
                   <h2 class="title is-3"><span class="dvima">What Can You Achieve with LLM2CLIP?</span></h2>