A clean, from-scratch PyTorch implementation of DINO (Self-Distillation with No Labels), the groundbreaking self-supervised learning method that discovers meaningful visual representations without any human annotations.

Description: This animation demonstrates the training progression of our self-implemented DINO model, showing improved feature extraction and segmentation capabilities over time.

Self-Supervised Learning (SSL) is a revolutionary paradigm where models learn representations from unlabeled data by creating their own supervision signals. Instead of relying on human-annotated labels, SSL systems generate "pretext tasks" that enable models to learn rich, transferable representations.

• Contrastive Learning: Learning by comparing similar and dissimilar examples
• Predictive Tasks: Predicting hidden or transformed parts of the input

1. Raw Unlabeled Data: Start with massive collections of images (e.g., from the internet)
2. Pretext Task: Create an artificial task where labels are automatically generated
3. Model Training: Train a model to solve this pretext task
4. Representation Learning: Discard the task-specific head and use the learned features for downstream tasks with minimal labeled data

DINO (DIstillation with NO labels) is a groundbreaking SSL algorithm that uses a simple yet powerful self-distillation framework to learn semantically meaningful image representations, even discovering object segmentation capabilities without any labels.

# Conceptual implementation of DINO's core mechanism
teacher_network = VisionTransformer()  # Processes global views
student_network = VisionTransformer()  # Processes local views

# Teacher weights are EMA of student weights
teacher_network.weights = EMA(student_network.weights)

# Knowledge distillation loss
loss = distillation_loss(
    student_network(local_view), 
    teacher_network(global_view)
)

The EMA update rule for a parameter vector is:

$$ \theta_{\text{teacher}} \gets m \times \theta_{\text{teacher}} + (1 - m) \times \theta_{\text{student}} $$

Where:

• $\theta_{\text{teacher}}$: Teacher model parameters

• $\theta_{\text{student}}$: Student model parameters

• $m$: Momentum coefficient (typically close to 1, e.g., 0.99, 0.996)

• Why EMA is Used in DINO

The teacher network provides consistent, slowly evolving targets for the student to learn from.

EMA helps avoid the trivial solution where both networks output constant representations.

The teacher acts as an ensemble of previous student models, capturing robust features.

• Global Views (e.g., 224×224 pixels) → Teacher Network
• Local Views (e.g., 96×96 pixels) → Student Network

• Sharpening: Temperature parameter in softmax produces "peaky" distributions
• Centering: Bias term prevents dimension domination
• Momentum Encoder: Stable targets via exponential moving average

# Setup Instructions

## 1. Clone the Repository
```bash
git clone https://github.com/basaanithanaveenkumar/object-detection-BBD.git
cd object-detection-BBD

mkdir -p data

python scripts/download_dataset.py

mv data/100k/val data/100k/valid

python scripts/convert_to_coco.py

This setup process:

1. Clones the object detection project repository
2. Creates the necessary directory structure
3. Downloads the BBD (Berkeley DeepDrive) dataset
4. Renames the validation directory to match expected conventions
5. Converts the BBD dataset format to standard COCO format for compatibility with object detection frameworks

Based on the original paper:
Emerging Properties in Self-Supervised Vision Transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin
arXiv | Official Implementation

MIT License - see LICENSE for details.

Contributions welcome! Please feel free to submit issues and pull requests.

⭐ If this project helps your research, please give it a star!