Merge branch 'main' of https://github.com/balaboom123/How2sign-Youtube-ASL-Dataset-Preprocess

Author: balaboom123

README.md

@@ -1,61 +1,76 @@
 # ASL Translation Data Preprocessing
 
-This repository provides a comprehensive solution for preprocessing American Sign Language (ASL) datasets, specifically designed to handle both How2Sign and YouTube ASL datasets. Our preprocessing pipeline streamlines the workflow from video acquisition to landmark extraction, making the data ready for ASL translation tasks.
+This repository follows the methodology described in the YouTube-ASL Dataset paper and provides a comprehensive solution for preprocessing American Sign Language (ASL) datasets, specifically designed to handle both **How2Sign** and **YouTube-ASL** datasets. Our preprocessing pipeline streamlines the workflow from video acquisition to landmark extraction, making the data ready for ASL translation tasks.
 
 ## Project Configuration
 
-All project settings are centrally managed through `conf.py`, offering flexible configuration options for video processing, dataset management, and feature extraction. The configuration file controls several key aspects:
+All project settings are centrally managed through `conf.py`, providing a single point of configuration for the entire preprocessing pipeline. Key configuration elements include:
 
-The system allows customization of video processing parameters, including frame skip rates and maximum frame limits, to optimize processing efficiency while maintaining data quality. It manages dataset paths and directories, ensuring organized data storage and retrieval. The configuration also specifies MediaPipe landmark indices for detailed capture of pose, face, and hand movements, essential for ASL translation. Additionally, it includes language preference settings for YouTube transcript collection, supporting various English language variants.
+- `ID`: Text file containing YouTube video IDs to process
+- `VIDEO_DIR`: Directory for downloaded videos
+- `TRANSCRIPT_DIR`: Storage for JSON transcripts
+- `OUTPUT_DIR`: Location for extracted features
+- `CSV_FILE`: Path for processed segment data
 
-## YouTube ASL Dataset Processing
+- `YT_CONFIG`: YouTube download settings (video quality, format, rate limits)
+- `LANGUAGE`: Supported language options for transcript retrieval
+- `FRAME_SKIP`: Controls frame sampling rate for efficient processing
+- `MAX_WORKERS`: Manages parallel processing to optimize performance
 
-The processing of YouTube ASL dataset follows a systematic three-step approach, ensuring comprehensive data preparation:
+- `POSE_IDX`, `FACE_IDX`, `HAND_IDX`: Selected landmark indices for extracting the most relevant points for sign language analysis
 
-### Step 1: Data Acquisition
+This centralized approach allows easy adaptation to different hardware capabilities or dataset requirements without modifying the core processing code.
+## How To Use?
+- **YouTube-ASL**: make sure the constant is correct in conf.py. Then, operate step 1 to step 3. 
+- **How2Sign**: download **Green Screen RGB videos** and **English Translation (manually re-aligned)** from How2Sign website. Put the directory and .csv file in the right path or amend the path in the conf.py. then, operate step 3 only.
+  
+### Step 1: Data Acquisition (s1_data_downloader.py)
+**Necessary Constants:**`ID`, `VIDEO_DIR`, `TRANSCRIPT_DIR`, `YT_CONFIG`, `LANGUAGE`  
+The script intelligently skips already downloaded content and implements rate limiting to prevent API throttling.
 
-This initial phase combines two parallel processes:
 
-The video downloader (`s1_video_download.py`) efficiently retrieves YouTube videos using yt-dlp, implementing smart rate limiting and quality control measures. It includes features for parallel fragment downloads and automatically skips previously downloaded content to prevent redundant processing.
 
-Simultaneously, the transcript collector (`s1_transcript_downloader.py`) obtains video transcripts through the YouTube Transcript API. This component handles multiple English language variants and saves the transcripts in a structured JSON format, while maintaining appropriate rate limits to ensure reliable data collection.
+### Step 2: Transcript Processing (s2_transcript_preprocess.py)
+**Necessary Constants:** `ID`, `TRANSCRIPT_DIR`, `CSV_FILE`  
+This step cleans text (converts Unicode characters, removes brackets), filters segments based on length and duration, and saves them with precise timestamps as tab-separated values.
 
-### Step 2: Transcript Enhancement
 
-The transcript processor (`s2_transcript_preprocess.py`) refines the raw transcript data into a format suitable for ASL translation. It performs sophisticated text normalization, including Unicode handling and ASCII conversion, while preserving semantic meaning. The system segments videos into overlapping chunks with precise timing information, generating well-structured CSV files containing the processed segments.
 
-### Step 3: Feature Extraction
+### Step 3: Feature Extraction (s3_mediapipe_labelling.py)
+**Necessary Constants:** `CSV_FILE`, `VIDEO_DIR`, `OUTPUT_DIR`, `MAX_WORKERS`, `FRAME_SKIP`, `POSE_IDX`, `FACE_IDX`, `HAND_IDX`  
+The script processes each video segment according to its timestamp, extracting only the most relevant body keypoints for sign language analysis. Results are saved as NumPy arrays.
 
-The landmark extraction system (`s3_mediapipe_labelling.py`) utilizes the MediaPipe Holistic model to capture detailed movement data. It processes video segments to extract comprehensive pose, face, and hand landmarks, leveraging parallel processing capabilities for efficient computation. The extracted features are stored as numpy arrays for subsequent analysis and translation tasks.
+## Dataset Introduction
 
-## How2Sign Dataset Processing
+### YouTube-ASL Dataset
+Video List: [https://github.com/google-research/google-research/blob/master/youtube_asl/README.md](https://github.com/google-research/google-research/blob/master/youtube_asl/README.md)  
+Paper: ["YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus" (Uthus et al., 2023)](https://arxiv.org/abs/2306.15162).
 
-For the How2Sign dataset, our system offers two specialized approaches for MediaPipe landmark extraction:
+If you use YouTube-ASL, please cite their associated paper:
 
-### Clip-Based Processing
+```
+@misc{uthus2023youtubeasl,
+  author = {Uthus, David and Tanzer, Garrett and Georg, Manfred},
+  title = {YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus},
+  year = {2023},
+  eprint = {2306.15162},
+  archivePrefix = {arXiv},
+  url = {https://arxiv.org/abs/2306.15162},
+}
+```
 
-The clip processor (`H2S_clip_mediapipe.py`) handles complete video clips in a single pass. It employs adaptive frame skipping to optimize processing speed while maintaining data quality. The system leverages parallel processing capabilities to handle multiple clips simultaneously, ensuring efficient resource utilization.
+### How2Sign Dataset
+Dataset: [https://how2sign.github.io/](https://how2sign.github.io/)  
+Paper: [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://openaccess.thecvf.com/content/CVPR2021/html/Duarte_How2Sign_A_Large-Scale_Multimodal_Dataset_for_Continuous_American_Sign_Language_CVPR_2021_paper.html)
 
-### Raw Video Processing
+If you use How2Sign, please cite their associated paper:
 
-The raw video processor (`H2S_raw_mediapipe.py`) takes a more granular approach, working with precise realigned timestamps from a CSV file. This method extracts landmarks for specific video segments, maintaining temporal accuracy while utilizing parallel processing for optimal performance.
-
-## Data Organization
-
-The system organizes processed data into clearly defined formats:
-- Video content is stored as MP4 files for optimal quality and compatibility
-- Transcripts are maintained in JSON format for easy parsing and manipulation
-- Segment information is organized in CSV files for straightforward analysis
-- Extracted landmarks are preserved as NumPy arrays (.npy files) for efficient processing
-
-## Technical Requirements
-
-The system relies on several key Python libraries:
-- OpenCV (cv2) for video processing
-- MediaPipe for pose and gesture recognition
-- NumPy for efficient numerical operations
-- Pandas for data manipulation
-- yt-dlp for video downloading
-- youtube-transcript-api for transcript retrieval
-
-This preprocessing pipeline creates a robust foundation for ASL translation tasks, ensuring high-quality data preparation while maintaining processing efficiency.
+```
+@inproceedings{Duarte_CVPR2021,
+    title={{How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language}},
+    author={Duarte, Amanda and Palaskar, Shruti and Ventura, Lucas and Ghadiyaram, Deepti and DeHaan, Kenneth and
+                   Metze, Florian and Torres, Jordi and Giro-i-Nieto, Xavier},
+    booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},
+    year={2021}
+}
+```