Benchmark Ideas

[curran]

In order to continuously improve this tool, we need benchmarks.

Here's a dump of ideas from a session with O1:

Here's a detailed plan to build out a robust suite of benchmarks for your AI-powered code editing tool, ensuring the generated code is reliable, correct, and thoroughly tested:

---

Step 1: Define Benchmark Structure

Create a standardized directory structure for each benchmark:

benchmarks/
├── problem-name/
│   ├── README.md
│   ├── prompt.md
│   ├── solution.js (or relevant language)
│   └── solution.test.js

• README.md: Describes the coding problem clearly.
• prompt.md: Contains the exact prompt you'll provide to the AI tool.
• solution.js: Ideal or reference implementation for the problem (human-written).
• solution.test.js: Unit tests validating the correctness of solutions.

---

Step 2: Problem Selection & Categorization

Categorize problems by complexity:

• Easy
  • FizzBuzz
  • Palindrome Checker
  • Factorial Calculation
  • Simple Array Operations
• Medium
  • Fibonacci Sequence
  • Flatten Nested Arrays
  • Balanced Parentheses Checker
  • Prime Number Generation
• Hard
  • Sudoku Solver
  • Pathfinding Algorithms (A*, Dijkstra's)
  • JSON Parser
  • Regex Parser and Matcher
• Real-world
  • REST API client implementation
  • CRUD operations with SQL or NoSQL databases
  • JSON data transformation
  • Handling asynchronous logic with promises and async/await

---

Step 3: Writing Robust Unit Tests

Ensure thorough unit test coverage using a testing framework like Jest or Vitest.

Example (factorial.test.js):

import { factorial } from './solution';

test('factorial of 0 is 1', () => {
  expect(factorial(0)).toBe(1);
});

test('factorial of 5 is 120', () => {
  expect(factorial(5)).toBe(120);
});

test('factorial of negative number throws error', () => {
  expect(() => factorial(-1)).toThrow('Negative input');
});

Unit tests should cover:

• Happy paths
• Edge cases
• Error handling scenarios
• Large input or scalability concerns

---

Step 4: Automate AI Integration and Test Execution

Create automation scripts to:

• Feed the AI tool the prompts (prompt.md) and capture its generated solution.
• Automatically run unit tests against the AI-generated code.

For example, a basic automation script in Node.js could be:

import { execSync } from 'child_process';
import fs from 'fs';

const benchmarks = fs.readdirSync('./benchmarks');

benchmarks.forEach(benchmark => {
  console.log(`Running benchmark: ${benchmark}`);

  const prompt = fs.readFileSync(`./benchmarks/${benchmark}/prompt.md`, 'utf-8');
  
  // Replace this with your AI integration logic
  const aiGeneratedCode = getAIGeneratedCode(prompt); 

  fs.writeFileSync(`./benchmarks/${benchmark}/ai_solution.js`, aiGeneratedCode);

  try {
    execSync(`vitest run ./benchmarks/${benchmark}/solution.test.js`);
    console.log(`${benchmark}: ✅ Passed`);
  } catch (e) {
    console.log(`${benchmark}: ❌ Failed`);
  }
});

(Replace getAIGeneratedCode with your actual AI integration function.)

---

Step 5: Track and Report Results

Capture results systematically:

• Maintain a summary CSV or JSON file.
• Include timestamps, pass/fail status, test coverage, and execution times.

Example JSON structure:

{
  "benchmarks": {
    "factorial": {
      "status": "pass",
      "runtime_ms": 32,
      "coverage": "100%",
      "timestamp": "2025-03-24T12:34:56Z"
    },
    "sudoku-solver": {
      "status": "fail",
      "error": "Timeout exceeded",
      "runtime_ms": 10000,
      "coverage": "N/A",
      "timestamp": "2025-03-24T12:36:02Z"
    }
  }
}

Use this data to analyze performance trends and identify areas for improvement.

---

Step 6: Continuously Expand and Improve

Regularly:

• Add more benchmarks covering diverse coding styles and real-world problems.
• Refine unit tests for better coverage.
• Introduce performance metrics like runtime and memory usage.

---

Recommended Next Steps:

• Choose initial problems and set up the first few benchmarks.
• Establish automation scripts to integrate the AI tool and run tests automatically.
• Track and iterate: Use gathered insights to improve the AI tool iteratively.

Tailoring benchmarks specifically for data visualization coding involves creating a set of coding tasks related to common visualization challenges, accompanied by clear prompts, generated code verification through rendering tests, and validations beyond traditional unit tests (e.g., visual regression testing).

Here's how you can approach it specifically for data visualization:

---

1. Define Visualization Benchmark Structure

Create each benchmark with a structure similar to:

benchmarks/
├── scatter-plot/
│   ├── README.md           # Problem description & dataset
│   ├── prompt.md           # Prompt for AI (clearly states goal & constraints)
│   ├── reference.js        # Correct human-written solution
│   ├── ai_solution.js      # AI-generated code (will be saved here automatically)
│   ├── dataset.csv/json    # Sample data for visualization
│   ├── tests/
│   │   ├── visual.test.js  # Visual regression / rendering tests
│   │   └── logic.test.js   # Data transformation unit tests
│   └── snapshots/          # Screenshots for visual regression testing

---

2. Curate Specific Data Visualization Problems

Categorize problems around popular and representative D3.js or other visualization tasks:

Basic visualizations:

• Bar Chart
• Scatter Plot
• Line Chart
• Pie Chart / Donut Chart
• Histogram

Intermediate:

• Heatmap
• Choropleth Map
• Grouped/Stacked Bar Charts
• Animated transitions between datasets
• Tooltip interactions

Advanced:

• Force-directed Graph Layouts
• Sankey Diagrams
• GeoJSON & Map projections
• Zoom & Pan Interactions
• Brushing & Linking across multiple charts

---

3. Craft High-Quality Visualization Prompts

A good prompt includes:

• Visualization goal (e.g., "render a scatter plot").
• Technical constraints (e.g., "use D3.js v7").
• Specific interactions or animations required.
• Input data description or explicit sample datasets.

Example prompt.md:

"Generate a scatter plot visualization using D3.js version 7, plotting 'GDP per Capita' on the x-axis against 'Life Expectancy' on the y-axis. Include axis labels, gridlines, and tooltips displaying detailed data upon hovering each point. Data is provided in dataset.csv."

---

4. Verification Beyond Traditional Unit Tests

Combine unit tests with visual testing:

Logic/Unit Tests (Jest/Vitest):

Verify data transformations and scale calculations.

import { computeScales } from '../reference.js';

test('x-scale domain computed correctly', () => {
  const data = [{ x: 1 }, { x: 5 }, { x: 10 }];
  const { xScale } = computeScales(data);
  expect(xScale.domain()).toEqual([1, 10]);
});

Visual Regression Tests (e.g., Playwright or Puppeteer):

Automatically take screenshots and compare visually to ensure correctness.

import { expect, test } from '@playwright/test';

test('scatter plot renders correctly', async ({ page }) => {
  await page.goto('/scatter-plot/index.html');
  const chart = await page.locator('#chart-container');
  expect(await chart.screenshot()).toMatchSnapshot('scatter-plot.png');
});

---

5. Automation Workflow for AI Integration & Testing

Automate a pipeline to:

• Run the AI code generator with the prompt.
• Save generated visualization code.
• Launch visualization in a headless browser environment.
• Execute both unit and visual regression tests.

Example workflow (high-level):

# Example pseudo-script for clarity
for benchmark in benchmarks/*; do
  prompt=$(cat $benchmark/prompt.md)
  ai_output=$(run_ai_tool "$prompt") 
  echo "$ai_output" > $benchmark/ai_solution.js

  # Launch visualization & run tests
  npm run visual-test -- $benchmark/tests/visual.test.js
  npm run logic-test -- $benchmark/tests/logic.test.js
done

---

6. Track Visualization Benchmarks Results

Maintain structured JSON/CSV for tracking performance and accuracy metrics. Include rendering success, visual regression differences, and interactivity tests.

Example tracking result:

{
  "scatter-plot": {
    "unit_tests_passed": true,
    "visual_regression_passed": false,
    "visual_diff_score": 0.08,
    "runtime_ms": 250,
    "timestamp": "2025-03-24T13:45:00Z"
  }
}

---

Recommended Next Steps:

• Define initial visualization benchmarks: Start with simple bar charts or scatter plots.
• Implement an automated visual regression testing workflow: Use Playwright or Puppeteer for screenshot comparisons.
• Iteratively enhance and refine benchmarks: Cover increasingly complex visualizations and interactions.

You're right—exact image matching isn't ideal for data visualization due to visual variability. Instead, leveraging human grading or Vision LLMs can offer more meaningful feedback.

Here's how you can practically set up these two alternate approaches:

---

Option 1: Human-in-the-loop Grading

Humans grade visualizations manually, guided by clear evaluation criteria.

Setup:

• Provide a grading rubric alongside generated visualizations.
• Grade aspects like clarity, correctness, readability, aesthetics, and interactivity.

Example Rubric:

Criterion	Score (0-3)	Notes
Data Correctness	3	All data points accurate & visible
Clarity	2	Labels present but overlapping slightly
Interactivity	1	Tooltip sluggish
Aesthetic & Style	2	Clean but colors hard to distinguish

Process:

• AI generates the visualization.
• Humans review each visualization against the rubric.
• Scores are aggregated and tracked over time.

Automation Aid:

• Set up a simple web-based dashboard to streamline grading.
• Integrate with Airtable, Google Sheets, or custom UI to quickly record scores.

---

Option 2: Vision-based LLM Automated Evaluation

Use Vision-based LLMs (like GPT-4V) to automate evaluations of visual outputs.

Example Prompt for GPT-4V:

Provide GPT-4V with the generated visualization and instruct it clearly:

“You're a data visualization expert tasked with reviewing this chart. Evaluate it based on data accuracy, clarity, readability, and aesthetic quality. Assign a score between 0 (poor) and 3 (excellent) for each category, and briefly justify your reasoning.”

Expected GPT-4V response format:

{
  "data_accuracy": { "score": 3, "reason": "All data points correctly represented." },
  "clarity": { "score": 2, "reason": "Axis labels clear, but some overlap present." },
  "readability": { "score": 3, "reason": "Font sizes and contrasts make it easy to read." },
  "aesthetics": { "score": 2, "reason": "Colors visually appealing but too similar in shade." }
}

Automation Workflow (GPT-4V API):

• Generate visualization code (AI output).
• Render the visualization and capture a screenshot.
• Call GPT-4V with the screenshot and get structured evaluation.
• Aggregate and store these automated scores.

Advantages:

• Rapid feedback cycle (minutes vs. human hours).
• Scalable evaluations across many benchmarks.
• Provides consistent, replicable scoring.

---

Recommended Hybrid Approach:

Combine automated Vision LLM evaluations with periodic human checks:

• Primary evaluation: Vision LLM automatically grades every visualization.
• Secondary evaluation (validation): Humans spot-check periodically to calibrate automated assessments and ensure validity.

Hybrid Workflow Example:

AI generates visualization
    |
    +---> Vision LLM scores automatically
                |
                +---> Periodic human validation & calibration
                          |
                          +---> Feedback loop improves prompt/automation

---

Next Steps:

• Decide which option fits your workflow best:
  • Human-in-the-loop: More accurate but slower.
  • Vision LLM: Fast, scalable, with moderate accuracy.
  • Hybrid: Balanced accuracy & scalability (recommended).
• Implement a quick prototype:
  • If Vision LLM: Set up GPT-4V integration for automatic visual evaluations.
  • If Human grading: Create a simple grading rubric and dashboard/UI.

Would you like help setting up a GPT-4V prompt, rubric, or technical details to kickstart your chosen evaluation process?

[curran]

This is a great source for the examples:

https://github.com/exercism/javascript/tree/main

It's used by the old Aider benchmarks:

https://aider.chat/docs/leaderboards/edit.html