deepseek is working

Author: naveenmiriyaluredhat

harness/USAGE_EXAMPLES.md

@@ -17,6 +17,18 @@ This document provides detailed examples for using the MLPerf Inference Harness
  python harness_main.py --model-category llama3.1-8b --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8  --dataset-path cnn_eval.json --dataset-name llama3.1-8b --server-config backendserver/simple.yaml --scenario Server --test-mode performance --batch-size 13368 --num-samples 13368 --output-dir TEST-SERVER --lg-model-name llama3_1-8b --server-target-qps 40
  ```
 
+### Sample command for Deepseek
+```bash
+ python harness_main.py --model-category deepseek-r1 --model deepseek-ai/DeepSeek-R1-0528 --dataset-path mlperf_deepseek_r1_dataset_4388_fp8_eval.pkl --dataset-name deepseek-r1 --scenario Offline --test-mode performance --batch-size 4388 --num-samples 4388 --output-dir <output_dir> --lg-model-name deepseek-r1 --server-config backendserver/deepseek.yaml --mlflow-experiment-name testing-stuff --mlflow-host ip --enable-metrics
+ ```
+
+### Sample command with MLflow auto-upload
+```bash
+ python harness_main.py --model-category llama3.1-8b --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8  --dataset-path cnn_eval.json --dataset-name llama3.1-8b --server-config backendserver/simple.yaml --scenario Server --test-mode performance --batch-size 13368 --num-samples 13368 --output-dir TEST-SERVER-2 --lg-model-name llama3_1-8b --server-target-qps 40 --mlflow-experiment-name testing-stuff --mlflow-host 150.239.115.202
+ ```
+
+**Note:** When `--mlflow-experiment-name` and `--mlflow-host` are provided, results are automatically uploaded to MLflow after the test completes successfully. No separate upload step is needed.
+
 ## Basic Usage
 
 ### Example 1: Simplest Case - Using Model Name Auto-Detection
@@ -81,86 +93,7 @@ python harness/harness_main.py \
 ### Example 5: DeepSeek R1 Model
 
 ```bash
-python language/deepseek-r1/harness_deepseek_r1.py \
-    --model deepseek-ai/DeepSeek-R1-0528 \
-    --dataset-path ./deepseek_dataset.pkl \
-    --dataset-name deepseek-r1
-```
-
-### Example 6: Using Model Config File
-
-```bash
-# Model config is auto-loaded from configs/models/llama3.1-8b.yaml
-python harness/harness_main.py \
-    --model llama3.1-8b \
-    --dataset-path ./cnn_eval.json
-```
-
-## Dataset Configuration
-
-### Example 7: Using Dataset Config File
-
-```bash
-# Specify exact config file to use
-python harness/harness_main.py \
-    --model meta-llama/Llama-3.1-8B-Instruct \
-    --dataset-path ./my_dataset.pkl \
-    --dataset-config-file configs/datasets/my-dataset.yaml
-```
-
-### Example 8: Overriding Column Names
-
-```bash
-# Override column mappings without creating config file
-python harness/harness_main.py \
-    --model my-model/MyModel \
-    --dataset-path ./dataset.pkl \
-    --input-column prompt \
-    --input-ids-column token_ids \
-    --output-column target
-```
-
-### Example 9: Combining Config and Overrides
-
-```bash
-# Use config file but override specific column
-python harness/harness_main.py \
-    --model meta-llama/Llama-3.1-8B-Instruct \
-    --dataset-path ./cnn_eval.json \
-    --dataset-config-file configs/datasets/llama3.1-8b.yaml \
-    --input-column custom_input  # Overrides config's input_column
-```
-
-### Example 10: Different Dataset Formats
-
-```bash
-# JSON dataset
-python harness/harness_main.py \
-    --model llama3.1-8b \
-    --dataset-path ./cnn_eval.json \
-    --dataset-name llama3.1-8b
-
-# Pickle dataset
-python harness/harness_main.py \
-    --model deepseek-r1 \
-    --dataset-path ./deepseek_dataset.pkl \
-    --dataset-name deepseek-r1
-
-# CSV dataset (needs config with column mappings)
-python harness/harness_main.py \
-    --model my-model \
-    --dataset-path ./my_dataset.csv \
-    --dataset-name my-dataset
-```
-
-## Endpoint Configuration
-
-### Example 11: Using Completions Endpoint (Default)
-
-```bash
-# Explicitly specify completions endpoint
-python harness/harness_main.py \
-    --model llama3.1-8b \
+model llama3.1-8b \
     --dataset-path ./cnn_eval.json \
     --api-server-url http://localhost:8000 \
     --endpoint-type completions

harness/backendserver/deepseek.yaml

@@ -0,0 +1,60 @@
+# Example YAML configuration for inference server
+# This file demonstrates how to configure an inference server
+
+# Backend type: 'vllm' or 'sglang'
+backend: vllm
+
+# Model name or path
+model: deepseek-ai/DeepSeek-R1-0528
+
+# Server port
+port: 8000
+
+# Output directory for server logs
+#output_dir: ./test_dir
+
+# Heartbeat configuration
+heartbeat_interval: 100      # Interval between heartbeat checks (seconds)
+heartbeat_timeout: 30      # Timeout for heartbeat checks (seconds)
+startup_timeout: 1200       # Timeout for server startup (seconds)
+
+# Debug mode - verify cleanup of server and child processes
+# Especially useful for tensor parallel/data parallel setups
+debug_mode: false          # Enable debug mode for process cleanup verification
+
+# Path to Python binary (default: 'python')
+#binary_path: python
+
+# Custom launch command (optional - overrides default command generation)
+# If specified, this command will be used instead of the backend's default command
+# launch_command:
+#   - python
+#   - -m
+#   - vllm.entrypoints.openai.api_server
+#   - --model
+#   - meta-llama/Llama-2-7b-hf
+#   - --port
+#   - "8000"
+
+# Environment variables to set
+env_vars:
+  CUDA_VISIBLE_DEVICES: "0,1,2,3,4,5,6,7"
+  OMP_NUM_THREADS: "16"
+  VLLM_STATS_LOG_INTERVAL: "5"
+  # Add any other environment variables here
+
+# Backend-specific configuration
+config:
+  # For vLLM
+  api_server_args:
+    - --tensor-parallel-size
+    - "8"
+    - --gpu-memory-utilization
+    - "0.9"
+    # Add any other vLLM arguments here
+  
+  # For SGLang
+  # server_args:
+  #   - --tp
+  #   - "1"
+

harness/data/dataset_processor.py

@@ -226,6 +226,16 @@ def _process_data(self):
         if hasattr(self, 'processed_data') and PANDAS_AVAILABLE:
             df = self.processed_data
 
+            # Print dataset column names and types
+            self.logger.info("=" * 80)
+            self.logger.info("DATASET COLUMNS AND TYPES")
+            self.logger.info("=" * 80)
+            for col_name, col_type in df.dtypes.items():
+                self.logger.info(f"  {col_name}: {col_type}")
+            self.logger.info(f"Total columns: {len(df.columns)}")
+            self.logger.info(f"Total rows: {len(df)}")
+            self.logger.info("=" * 80)
+            
             # Extract input column
             if self.input_column in df.columns:
                 self.input = df[self.input_column].tolist()
@@ -255,6 +265,33 @@ def _process_data(self):
 
         elif hasattr(self, 'raw_data'):
             # Handle raw dict/list data
+            # Print dataset column names and types for raw data
+            self.logger.info("=" * 80)
+            self.logger.info("DATASET COLUMNS AND TYPES")
+            self.logger.info("=" * 80)
+            if isinstance(self.raw_data, dict):
+                # Dict with lists as values
+                for key, value in self.raw_data.items():
+                    if isinstance(value, list) and len(value) > 0:
+                        sample_type = type(value[0]).__name__
+                        self.logger.info(f"  {key}: list[{sample_type}] (length: {len(value)})")
+                    else:
+                        self.logger.info(f"  {key}: {type(value).__name__}")
+                self.logger.info(f"Total columns: {len(self.raw_data)}")
+            elif isinstance(self.raw_data, list):
+                # List of dicts
+                if len(self.raw_data) > 0:
+                    first_item = self.raw_data[0]
+                    if isinstance(first_item, dict):
+                        for key, value in first_item.items():
+                            self.logger.info(f"  {key}: {type(value).__name__}")
+                        self.logger.info(f"Total columns: {len(first_item)}")
+                        self.logger.info(f"Total rows: {len(self.raw_data)}")
+                    else:
+                        self.logger.info(f"  (list item type: {type(first_item).__name__})")
+                        self.logger.info(f"Total items: {len(self.raw_data)}")
+            self.logger.info("=" * 80)
+            
             if isinstance(self.raw_data, dict):
                 # Dict with lists as values
                 self.input = self.raw_data.get(self.input_column, [])