ChartAnchor: Chart Grounding with Structural-Semantic Fidelity and Data Recovery

Xinhang Li, Jingbo Zhou, Pengfei Luo, Yixiong Xiao, Tong Xu
Baidu Research, USTC
{lixinhang, zhoujingbo, luopengfei, xiaoyixiong}@baidu.com, [email protected]

(2026)

Abstract.

Recent advances in multimodal large language models (MLLMs) highlight the need for benchmarks that rigorously evaluate structured chart comprehension. Chart grounding refers to the bidirectional alignment between a chart’s visual appearance and the structured semantics. This task requires models to produce a symbolic specification that faithfully captures the chart’s visual and structural intent, while also recovering the underlying tabular data with precise values and relationships. Chart grounding directly reflects a model’s capabilities in numerical reasoning, multimodal alignment, and structural reconstruction, and has several important applications in real-world scenarios. Existing benchmarks, constrained by narrow chart diversity, isolated tasks, and incomplete evaluation frameworks, fail to holistically assess grounding. To address this, we propose ChartAnchor, a comprehensive benchmark of 8k+ chart-table-code triples spanning 30 chart types drawn from diverse real-world and augmented sources. ChartAnchor introduces two complementary tasks: chart-to-code generation (synthesizing executable code to replicate charts) and controlled chart-to-table reconstruction (extracting exact data with predefined headers), enabling cross-validation of visual and numerical fidelity. A multi-level evaluation framework integrates semantic validation, stylistic analysis, and perceptual metrics to assess both structural and content-level correctness. Extensive experiments on MLLMs reveal critical limitations in numerical precision and code synthesis, emphasizing the need for structured reasoning beyond surface-level perception. By unifying symbolic and data-driven grounding, ChartAnchor establishes a rigorous foundation for chart grounding, offering meaningful insights for advancing MLLMs in scientific, financial, and industrial domains. Our code and data are available at https://github.com/immortal5655/ChartAnchor.

Chart Grounding; Chart Understanding; MLLMs; Multimodal Reasoning.

^†^†copyright: acmlicensed^†^†journalyear: 2026^†^†doi: XXXXXXX.XXXXXXX^†^†conference: ACM SIGKDD Conference on Knowledge Discovery and Data Mining; June 03–05, 2026; Jeju Island, South Korea^†^†isbn: 978-1-4503-XXXX-X/2018/06

1. Introduction

“A picture is worth a thousand words.” Charts exemplify this principle by converting complex datasets into intuitive visual representations, enabling rapid and effective communication of quantitative insights. As a result, data visualizations are essential tools across domains such as science, finance, journalism, and public policy. With the rise of multimodal large language models (MLLMs), there is increasing interest in extending their capabilities to understand and reason about charts—by jointly interpreting visual features and underlying symbolic structures.

One of the core capabilities of MLLMs in chart understanding and reasoning lies in the challenge of chart grounding—aligning a chart’s visual presentation (marks, axes, layout, colours, chart type) with its structured semantics (tabular data, scales). A grounding model must therefore (i) recover the exact data values from the image (chart → table) and (ii) capture the visual–structural encoding that turns those values into graphics (chart → code). Although such structural encodings could be summarised in natural language, we formalise them as executable plotting code: code provides an unambiguous, machine-verifiable specification of marks, scales, and layout, whereas free-text descriptions are often underspecified and cannot guarantee identical rendering. Mastery of both tasks evidences genuine numerical reasoning, spatial-layout comprehension, and multimodal alignment.

Chart grounding has significant applications in real-world scenarios. Fundamental use cases include the extraction of numerical values from charts for in-depth data analysis, as well as the regeneration of chart code to facilitate the creation of similar visualizations using new data. More importantly, chart grounding enables MLLMs to develop a comprehensive understanding of charts, providing a foundation for more advanced tasks. Potential applications include: (1) chat-based chart modification, in which users issue natural language instructions to alter a chart’s visual presentation or underlying data; (2) chart-grounded retrieval, where detailed chart code and data derived from charts can be indexed for downstream tasks such as retrieval-augmented generation (RAG); and (3) multi-chart reasoning, where grounding information from multiple charts and natural language instructions are integrated to support complex reasoning tasks. Therefore, it is crucial to assess the chart grounding capabilities of MLLMs—a important dimension that has not yet been fully explored.

While recent chart-related benchmarks such as visual QA (e.g., ChartQA(masry2022chartqa), PlotQA(methani2020plotqa)) and the summarization (e.g., Chart-to-Text(kantharaj2022chart), ChartSumm (rahman2023chartsumm)) have advanced chart understanding, they evaluate only unstructured outputs—offering limited insight into a model’s ability to recover symbolic or structured content. Tasks like chart-to-table and chart-to-code begin to address structured aspects, but each only tells part of the story. Chart-to-table benchmarks assess data recovery but ignore style and structure. Conversely, chart-to-code (e.g. ChartMimic(yang2024chartmimic), Plot2Code(wu2024plot2code)) benchmarks are predominantly focus on visual appearance, including layout, styling, and chart-surface text. While they may assess textual accuracy on the chart surface, these benchmarks still judge success primarily through visual similarity and do not enforce consistency with the underlying structured data, which poses serious risks in data-sensitive scenarios.

Critically, most tasks in chart understanding are evaluated in isolation. This fragmented approach allows models to exploit superficial shortcuts: for instance, question-answering systems may guess answers based solely on surface-level cues, without genuinely interpreting the underlying chart structure. While code-generation models may reproduce a chart’s appearance while silently altering the underlying data. Without unified evaluation, such partial successes can mask critical failures. Compounding the issue, existing benchmarks are limited in scope — they cover only a narrow range of chart types, exclude domain-specific formats, and depend on a single plotting library. This narrow design fails to capture the diversity of real-world visualizations and lacks robust, comprehensive evaluation metrics.

To bridge these gaps, we propose ChartAnchor, a large-scale benchmark specifically designed for comprehensive chart grounding. It comprises 8,068 chart–table–code triples spanning 30 diverse chart types, sourced from over 6,533 real-world examples manually collected by us, along with 1,535 augmented instances derived from existing datasets. The images and corresponding code are drawn from a selection of popular plotting libraries, reflecting diverse visual styles and implementation patterns found in real-world settings. Crucially, it introduces two complementary tasks designed to probe different facets of grounding. The first task, Chart-to-Code Generation, requires the model to synthesize executable Python code use indicated plotting library that replicates a given chart. This assesses the model’s understanding of stylistic and structural components—such as axis configuration, data-to-mark mapping, and layout decisions—while implicitly requiring correct data recovery. The second task, Controlled Chart-to-Table Reconstruction, focuses explicitly on data accuracy: given column headers, the model is required to extract the tabular data from the chart image, isolating numerical precision and structural alignment. By providing headers, the task eliminates label ambiguity and allows for a more focused evaluation of data fidelity. Beyond task design, ChartAnchor introduces a four-dimensional evaluation framework tailored to assessing and guiding the development of MLLMs for chart grounding and understanding. It provides the first unified diagnostic system that jointly evaluates functional validity (execution pass rate), visual rigor (verification of chart type, color, text, and layout), semantic data fidelity (structured tuple matching), and perceptual consistency (CLIP-based semantic alignment). Rather than relying on limited or isolated metrics, this framework enables comprehensive assessment of both code-level reasoning and data-level understanding, laying a foundation for building multimodal models that integrate computational precision with visual and semantic coherence.

2. Related Work

MLLM. Building on the success of Large Language Models (LLMs) (touvron2023llama; bai2023qwen), Multimodal Large Language Models (MLLMs) extend their capabilities to jointly process visual, video, and textual data, showing state-of-the-art performance across diverse scenarios. The current landscape can be divided into two branchs: 1) Open-source models—such as mPLUG-Owl2 (ye2024mplug), Qwen2.5-VL(wang2024qwen2), CogVLM (hong2024cogvlm2), and the more recent InternVL3 (zhu2025internvl3) employ efficient architectures through frozen vision encoders, lightweight adapters, or mixture-of-experts (MoE) designs. These models target specialized capabilities such as long-context processing, document analysis, and multilingual support, facilitating deployment in resource-limited scenarios. 2) Closed-source models (e.g., GPT-4V (yang2023dawn), GPT-4o (openai2024) Gemini-1.5-Pro (team2024gemini), Claude-3 (anthropic2024haiku)), leverage vast private datasets and extreme computational budgets to fuse text, image, audio, and even video streams within a single architecture. While their internal parameters are not publicly available, these systems establish widely adopted standards for academic research and industry applications, advancing evaluation methodologies and safety studies.

Chart Benchmark. As chart comprehension emerges as a key challenge for multimodal reasoning in MLLMs, a variety of benchmarks have been developed to assess different capabilities. Chart-to-Text (kantharaj2022chart) and ChartSumm (rahman2023chartsumm) evaluate summarization via structured textual descriptions. For visual question answering, ChartQA (masry2022chartqa), FigureQA (kahou2017figureqa), DVQA (kafle2018dvqa) and MultiChartQA (zhu2024multichartqa) provide multi-scale challenges spanning element localization and trend extrapolation. Benchmarks like ChartMimic (yang2024chartmimic), Plot2Code (wu2024plot2code), and Design2Code (si2024design2code) assess models’ abilities to generate accurate and executable code from chart images, highlighting the challenges in aligning visual understanding with code generation. While recent benchmarks such as ChartBench (xu2023chartbench) and ChartX (xia2024chartx) move beyond isolated tasks by incorporating diverse reasoning and perception challenges, they do not explicitly unify visual understanding with structured semantic outputs. Moreover, existing datasets are limited by their diversity in chart types, fixed rendering libraries, and metric calculations.

In contrast to existing benchmarks, we introduce ChartAnchor, a new benchmark designed to advance the field by formalizing the chart grounding task. Chart grounding is a unified task that jointly maps visual charts to both executable code and tabular data, effectively bridging chart-to-code generation and controlled table extraction within a single, interpretable framework. Despite its practical relevance, this task has been largely overlooked by existing benchmarks. ChartAnchor includes 30 diverse chart types, over 8,000 test instances, and rich image–table–code tuples rendered from multiple charting libraries, enabling more robust and generalizable evaluation of MLLMs.

Refer to caption — Figure 1. Chart Type Distribution in ChartAnchor.

3. The ChartAnchor Benchmark

To support robust evaluation of chart grounding, this section introduces two core tasks: chart-to-code generation and controlled chart-to-table reconstruction. We construct a diverse corpus of chart–table–code triplets by leveraging both existing datasets and real-world sources. The overall distribution of chart types in ChartAncor is shown in Fig. 2. We then present our parametric code generation and augmentation pipeline, followed by a filtering process that combines automated validation and expert review to ensure high-quality, semantically faithful data.

3.1. Task Definitions

As depicted in Fig. 2, to support robust evaluation of chart grounding, we define two core tasks in our benchmark: chart-to-code generation and controlled chart-to-table reconstruction. Each task is grounded in the chart–table–code triplets we collect and reflects distinct practical scenarios relevant to real-world chart understanding and visual reasoning.

Chart-to-Code Generation. This task challenges a model to generate executable Python plotting code from a chart image. Given an input image $I$ , the model must produce a valid script that accurately reconstructs the chart’s visual and structural elements, including the chart type, underlying data, axes, titles, legends, color scheme, and layout. The generated code should be both syntactically correct and semantically faithful, yielding a chart that closely matches the input. This task assesses the model’s capabilities in visual abstraction, symbolic reasoning, and code synthesis.

Controlled Chart-to-Table Reconstruction. This task focuses on recovering the underlying tabular data from a chart image under constrained header supervision. Given an input chart image $I$ and a predefined set of column headers $\mathcal{H}=\{h_{1},h_{2},\dots,h_{n}\}$ , the model is required to generate a table $\mathcal{T}=[r_{1},r_{2},\dots,r_{m}]$ , where each row $r_{j}$ assigns numerical values to the corresponding headers. This task evaluates the model’s ability to perform fine-grained numerical reasoning and precise visual interpretation. The inclusion of headers removes label ambiguity, enabling a more targeted and reliable evaluation of data fidelity.

3.2. Data Collection

In this section, we briefly present our data collection pipeline for building a high-quality chart–table–code corpus. ChartAnchor comprises 8,068 validated samples, with 1,535(19%) from augmented existing datasets and 6,533(81%) curated from real-world sources. Detailed procedures are described in Appendix A.

3.2.1. Data Source

As illustrated in Fig 3, we collect a diverse set of chart–table–code triplets from two main sources:

•

Existing datasets. We collect chart–table pairs from four well-established datasets: PlotQA(methani2020plotqa), DVQA(kafle2018dvqa), FigureQA(kahou2017figureqa) and Vistext(tang2023vistext). Since these datasets do not provide the corresponding Python code, we construct scripts to regenerate the chart images using the provided tables and chart metadata. To enhance diversity and prevent data leakage, we further apply systematic augmentations to various visual and semantic attributes during code generation.
•

Newly curated dataset. We curate a large collection of charttable-code triples from publicly web sources, where charts are created and contributed by real users across a wide range of domains.

3.2.2. Code Generation and Augmentation for Existing Data-
sets.

To address the lack of source code in existing chart datasets, we propose a parametric pipeline that translates chart metadata into executable plotting scripts. The pipeline consists of three main stages: semantic mapping of chart elements to primitives in visualization libraries, parameterization of visual attributes such as colors, fonts, and layout, and systematic augmentation through controlled style variations. These augmentations include changes to color schemes, marker types, axis scales, legend positions, font settings, and label orientations.

3.2.3. Filtering.

Our filtering process consists of two stages:

Automated Filtering. We apply a four-stage filtering process to ensure data quality and consistency:

(a) Completeness Check: Remove all incomplete data (table, chart, code) triples.
(b) Structural Filtering: Exclude geospatial maps, rasterized tables, and overly decorated charts that lack recoverable structure.
(c) Deduplication: We compute a compact feature signature for each sample based on data characteristics, including the number of entries, value counts, mean, standard deviation, min/max values, and the number of unique values. These features are encoded as string signatures for fast comparison, and duplicates are filtered by exact signature matching.
(d) Executability Check: We execute all Python scripts and discard any triples with runtime errors or failed rendering.

Human Filtering. Each chart–table–code triple was independently reviewed by experts experienced in deep learning and with hands-on experience in Python programming, data visualization, and chart scripting. Samples were grouped by chart type and filtered within each group to ensure consistency across chart families. Using a standardized rubric, reviewers evaluated each triple along three dimensions: semantic accuracy, visual clarity, and stylistic diversity, and assigned per-triple scores accordingly. Each sample was reviewed by two experts under a blind, rotating setup. In cases of major disagreement, a third expert served as an adjudicator to ensure annotation reliability and consensus.

Type	Scatterpolar	Scatter3d	Line3d	Pie	Barpolar	Mesh3d	Violin	Line	Histogram2d	Bar
Num	653	223	62	700	350	103	214	617	174	659
Type	Box	Scatterternary	Waterfall	Heatmap	Scatter	Cone	Surface	Histogram	Carpet	Treemap
Num	423	142	111	372	675	44	84	482	23	121
Type	Parcoords	Funnelarea	Funnel	Sankey	Candlestick	Contour	Sunburst	Histogram2dcontour	Areachart	Ohlc
Num	71	97	175	231	116	246	283	142	425	50

Table 2. Chart counts across types.

3.3. Dataset Analysis

Tab. 1 and Fig. 4 summarize the structural and visual diversity of ChartAnchor: 8,068 charts with a range of resolutions, averaging at 3,346×2,266px, brightness variability ( $\sigma$ = 42.35), and entropy ( $\mu$ = 1.41). Code length spans from concise to complex scripts (Fig. 4a), while visual metrics show moderate entropy (Fig. 4b), diverse brightness/contrast (Fig. 4c), and broad color-structural distributions (Fig. 4d), reflecting its real-world relevance by a balance between informativeness and realism. Token counts are computed using the GPT-4 tokenizer for tokenization.

Dataset

Types

Test

Set

Input

Format

Output

Format

Multi-task

Eval

Visual

Open

Source

Plot

Library

Data

Eval

Full

Ground

Chart Benchmarks

ChartQA(masry2022chartqa)

10k

I+NL

✗

✓

✗

PlotQA(methani2020plotqa)

34k

I+NL

✗

✓

✗

Chart-to-text(kantharaj2022chart)

44k

I+NL

✗

✓

✗

OpenCQA(kantharaj2022opencqa)

1.2k

I+NL

✗

✓

✗

ChartSumm(rahman2023chartsumm)

84k

I+NL

✗

✓

✗

Charxiv(wang2024charxiv)

1.32k

I+NL

✓

✗

MMC(liu2023mmc)

I+NL

✓

✗

ChartX(xia2024chartx)

I+NL

✓

✗

ChartBench(xu2023chartbench)

2.1k

I+NL

✓

✗

Code Generation Benchmarks

HumanEval(chen2021evaluating)

164

Code

✓

MBPP(austin2021program)

500

NL+Code

Code

✓

MMCode(li2024mmcode)

263

I+NL

Code

✓

MatPLotBench(yang2024matplotagent)

100

Code

✓

Mixture

✗

PLot2Code(wu2024plot2code)

132

I+NL

Code

✓

Singal

✗

ChartMimic(yang2024chartmimic)

4.8k

I+NL

Code

✓

Singal

✗

Design2Code(si2024design2code)

484

I+NL

Code

✓

Singal

✗

ChartAnchor

8.1k

I+NL

Code+NL

✓

Mixture

✓

Table 3. Comparison of benchmarks for chart understanding and code generation. “I” = image, “NL” = natural language. “Visual” assesses unannotated chart reasoning. “Plot Library” denotes the use of multiple plotting libraries in the chart-to-code task, while “Data Eval” indicates whether data fidelity is evaluated within the chart-to-code task.

As shown in Tab. 2, ChartAnchor encompasses 30 visualization families. Beyond canonical 2D Cartesian plots (e.g., bar, line, scatter), it cover a wide variety of chart types: (i) advanced 3D charts such as scatter3d, surface, mesh3d, and line3d; (ii) hierarchical/flow diagrams including sunburst, sankey, treemap, and funnelarea; (iii) polar and ternary variants (e.g., scatterpolar, barpolar); and (iv) finance–specific charts such as candlestick and ohlc. The overall distribution reflects realistic usage patterns, with frequent types (bar, line, pie) having 600–700 samples, mid-frequency ones 200–400, and fewer complex charts. By combining curated benchmarks with real-world user code, ChartAnchor exposes models to a diverse set of plotting libraries (e.g., matplotlib, plotly), varied scripting idioms, and domain-specific styling practices, thereby offering a robust and practical benchmark for evaluating the chart grounding capabilities of MLLMs. More analysis are in Appendix B. We also provide representative chart examples from each category, highlighting visual diversity and structural complexity, are illustrated in Appendix G.

3.4. Comparition with other benchmarks

Table 3 presents a comprehensive comparison between our proposed benchmark and a wide range of existing datasets related to chart understanding and code generation. We evaluate them along key dimensions, including input/output modalities, task diversity, support for visual reasoning, openness, plotting library variety, data fidelity assessment, and the completeness of symbolic supervision.

Earlier chart benchmarks such as ChartQA, PlotQA, and Chart-to-Text primarily focus on natural language tasks like question answering and summarization, without structured protocols to evaluate bidirectional alignment between chart visuals and their underlying data or specifications. More recent multi-task datasets such as ChartArxiv and ChartBench broaden the evaluation scope, yet still omit code-based grounding, limiting their ability to assess executable understanding of visual semantics.

Code generation benchmarks like HumanEval and MBPP are designed for text-to-code synthesis and lack any visual input, making them unsuitable for evaluating multimodal alignment. Multimodal code-focused datasets like Plot2Code and ChartMimic introduce image-to-code generation but do not explicitly assess numerical fidelity or data reconstruction accuracy.

In contrast, ChartAnchor is explicitly designed to evaluate comprehensive chart grounding. It uniquely integrates the following features: (1) High diversity with 30 chart types and multiple plotting libraries to support varied symbolic mappings; (2) Fidelity checks for both stylistic rendering and precise data recovery; (3) Support for unannotated chart images to facilitate unsupervised visual reasoning; (4) Chart–table–code supervision for rigorous evaluation of symbolic, visual, and structural alignment.

4. Evaluation Metrics

While existing chart-to-code benchmarks typically rely on basic metrics such as execution success or superficial visual similarity, they fail to capture semantic fidelity, particularly the correctness of underlying chart data. To address this, we introduce a multi-level evaluation framework that jointly examines the functional correctness, visual integrity, and data faithfulness of model outputs. This enables more precise assessment of chart grounding performance in real-world scenarios.

Functional Validity. Pass Rate measures the proportion of model outputs that execute or parse without errors—i.e., valid chart-rendering code or well-formed tables—indicating baseline reliability.

Visual Structure Consistency. To go beyond functional success, we perform fine-grained evaluation of visual structure through four key aspects extracted directly from the rendered chart objects:

•

1) Textual Components Match. We extract textual components—titles, axis labels, legends, annotations—and compare them against reference charts to verify semantic and positional consistency.

Model

Pass Rate

Visual Structure Consistency

Data Fidelity

Clip Score

Over all

Text

Color

Type

Layout

Legend

Title

Axis

Label

Annos

Avg.

Proprietary Multimodal Large Language Models

GPT4o

91.88

63.53

72.83

67.20

76.36

69.98

34.64

70.06

80.73

35.85

86.88

63.02

Claude-3-7-Sonnet

78.60

54.97

69.11

67.19

70.57

65.46

40.79

59.89

76.53

30.30

94.11

61.18

Open-Source Multimodal Large Language Models

InternVL3-2B

28.97

18.75

17.41

17.63

20.88

18.67

8.98

16.01

21.55

13.56

26.95

17.62

Qwen2.5-VL-3B-Instruct

48.07

28.24

31.80

26.86

39.44

31.58

12.29

27.01

40.19

16.71

43.98

28.63

Gemma-3-4B-it

66.20

42.29

38.72

36.41

57.50

43.73

18.74

33.53

60.46

24.21

60.98

40.28

DeepSeek-VL-7B

32.05

17.59

20.41

17.62

23.19

19.70

9.56

17.37

24.83

7.74

31.34

18.42

LLaVA-v1.6-Mistral-7B

15.32

8.74

8.37

4.96

4.91

6.75

6.03

7.79

5.05

4.76

13.94

7.39

Qwen2.5-VL-7B-Instruct

67.59

41.75

41.46

43.60

61.90

47.18

18.45

41.83

64.30

27.85

62.48

43.68

MiniCPM-V-2.6-8B

26.87

16.76

16.88

14.21

16.18

16.01

7.91

13.77

16.92

12.95

24.87

15.40

InternVL3-9B

69.19

44.75

47.48

45.06

59.57

49.20

22.06

40.72

66.20

29.20

64.34

45.29

GLM-4V-9B

46.04

26.55

27.86

24.55

43.78

30.68

12.02

23.32

45.69

15.51

42.60

28.30

CogVLM2-Llama3-Chat-19B

7.82

7.12

6.03

4.79

1.25

4.80

3.77

5.61

1.21

3.52

7.17

4.35

InternVL3-14B

80.60

52.80

55.69

54.90

75.12

59.63

26.13

53.08

80.63

30.59

75.49

54.26

Qwen2.5-VL-32B-Instruct

67.59

42.71

48.69

48.91

73.55

53.47

23.56

48.29

82.08

21.14

64.04

48.76

Table 4. Comprehensive performance comparison of proprietary and open-source MLLMs on the Chart-to-Code generation task. For data fidelity, we report IoU score under a slight tolerance stage. As our collected matplotlib images do not include annotations by design, they are excluded from the annotation metric evaluation.

•

2) Color Fidelity. We quantify perceptual color differences using the CIEDE2000 (sharma2005ciede2000) metric in the CIE Lab color space, which aligns with human vision sensitivity. This evaluation covers both static elements (background, axes) and dynamic color mappings (data series, gradients). For multi-color comparisons (e.g., categorical series), we apply the Hungarian algorithm to optimally pair generated and reference colors, minimizing the total perceptual deviation ( $\Delta E_{00}$ ) across matched pairs. across matched pairs. This ensures semantic alignment of color associations while preserving numerical fidelity.
•

3) Chart Type Identification. Chart types indicate the structural intent of a visualization. We determine the type distribution for both generated and reference charts by identifying the type of each visual element, and measure accuracy through distributional comparison.
•

4) Layout Alignment. We evaluate the presence, number, size, and arrangement of subplots to ensure structural correctness in multi-panel charts.

Semantic Data Fidelity.

To evaluate whether models can faithfully recover the underlying chart data, we introduce a data-level fidelity metric, applicable to both chart-to-code and controlled chart-to-table tasks. To the best of our knowledge, this is the first work to introduce structured data fidelity evaluation in the chart-to-code setting, moving beyond functional or visual correctness to explicitly verify the semantic integrity of generated chart data. In the chart-to-code task, we first parse the generated figure objects and dispatch them to type-specific extractors $\text{Extract}_{T}$ , where each $T\in\mathcal{T}$ corresponds to a supported chart type (e.g., line, bar, candlestick). Each extractor retrieves the relevant data fields—such as $(x,y)$ for line or $(x,\text{low},\text{high},\text{open},\text{close})$ for candlestick charts—and formats each data point as a normalized tuple $\tau_{i}=(n,x_{i},y_{i},\ldots)$ , where $n$ is the trace name and the remaining elements are type-specific field values. The final structured form is a set $\mathcal{L}_{T}=\{\tau_{1},\tau_{2},\ldots,\tau_{n}\}$ . In the controlled chart-to-table task, each table row is similarly treated as a tuple $\tau_{i}=(t_{1},t_{2},\ldots,t_{m})$ , resulting in a comparable structured set $\mathcal{L}_{T}$ , allowing both tasks to be evaluated under a unified tuple-based framework.

To assess prediction quality, we adopt a matching scheme inspired by the Structuring Chart-oriented Representation Metric (SCRM) (xia2023structchart). For each predicted tuple $p$ and ground-truth tuple $q$ , we compare corresponding fields $p_{i}$ and $q_{i}$ , where $i=0,1,\dots,n{-}1$ and $n$ is the tuple length. String fields are evaluated using edit distance $J(p_{i},q_{i})$ , and numerical fields using relative error $e(p_{i},q_{i})$ . A tuple is considered correct only if all fields satisfy their respective tolerance, with three tolerance levels: strict ( $J\leq 0$ or $e\leq 0$ ), slight ( $J\leq 3$ or $e\leq 0.05$ ), and high ( $J\leq 5$ or $e\leq 0.10$ ).

We report precision, recall, F1, and Intersection over Union (IoU) under a given tolerance level as:

P_{tol}=\frac{|M|}{|\text{Pred}|},\quad R_{tol}=\frac{|M|}{|\text{GT}|},\quad

\text{F1}_{tol}=\frac{2\cdot P_{tol}\cdot R_{tol}}{P_{tol}+R_{tol}},\quad\text{IoU}_{tol}=\frac{|M|}{|\text{Pred}|+|\text{GT}|-|M|}

Here, $|M|$ is the number of matched tuples under tolerance, and $|\text{Pred}|$ , $|\text{GT}|$ are the counts of predicted and ground truth tuples, respectively. This metric provides a principled and fine-grained assessment of semantic alignment and numerical fidelity, offering a robust measure of structured data reconstruction across both tasks.

Perceptual Similarity. To bridge the gap between syntactic correctness and human-like perception, We employ CLIPScore to evaluate image semantic consistency through embedding alignment. This complements structured evaluations by quantifying conceptual fidelity grounded in visual cognition.

5. Evaluation

In this section, we present a comprehensive evaluation of various MLLMs using our ChartAnchor benchmark. Our assessment includes both state-of-the-art proprietary (closed-source) models and representative open-source MLLMs, offering a holistic comparison of their performance.

Model	Pass Rate	P	R	F1	P	R	F1	P	R	F1
Proprietary Multimodal Large Language Models
GPT4o	97.12	10.74	8.44	8.65	37.71	23.30	24.69	60.42	35.44	37.72
Claude-3-7-Sonnet	94.85	12.75	11.02	10.86	44.64	29.43	30.52	58.12	37.11	38.62
Open-Source Multimodal Large Language Models
InternVL3-2B	55.16	5.84	3.86	4.08	22.18	10.88	11.98	51.04	18.29	21.19
Qwen2.5-VL-3B-Instruct	92.18	9.11	6.13	6.51	36.70	17.28	19.12	61.01	27.43	30.73
Gemma-3-4B-it	96.23	7.35	5.38	5.52	32.05	18.33	19.72	48.89	26.49	28.60
DeepSeek-VL-7B	98.34	1.72	0.36	0.49	5.47	0.90	1.33	26.95	4.79	7.02
LLaVA-v1.6-Mistral-7B	76.11	4.57	1.66	2.02	22.22	6.33	7.72	44.38	12.15	14.89
Qwen2.5-VL-7B-Instruct	95.05	12.17	8.18	8.67	48.45	23.75	26.35	69.58	30.96	34.90
MiniCPM-V-2.6-8B	79.66	5.07	3.03	3.30	26.80	11.07	12.51	53.36	18.98	22.09
InternVL3-9B	88.43	11.42	8.79	8.95	44.54	25.36	26.98	63.49	34.47	36.96
GLM-4V-9B	91.93	7.14	4.86	5.12	27.42	13.17	14.50	59.50	24.64	27.70
InternVL3-14B	66.18	8.19	7.47	7.28	29.27	22.30	22.81	40.90	29.67	30.49
CogVLM2-Llama3-Chat-19B	78.41	2.43	0.87	1.06	10.28	3.92	4.56	25.84	8.37	10.36
Qwen2.5-VL-32B-Instruct	97.83	13.05	9.87	10.10	48.65	28.94	30.99	68.14	37.97	41.04

Table 5. Comparison of model performance on the controlled chart-to-table task. Pass Rate indicates the proportion of examples for which the model produced a parsable table. P, R, and F1 denote precision, recall, and F1-score, respectively. Colored markers represent different tolerance levels: for strict, for slight, and for high.

5.1. Benchmarking Models

We evaluate 14 MLLMs. For proprietary models, we include two state-of-the-art enterprise-level models: GPT-4o (openai2024) and Claude-3-7-Sonnet (claude-sonnet), which represent the forefront of commercially deployed multimodal AI systems. On the open-source side, we consider a diverse set of architectures, primarily vision-language models (VLMs), across various parameter scales. In the compact category (under 7B parameters), we evaluate InternVL3-2B(zhu2025internvl3), Qwen2.5-VL-3B-Instruct(yang2024qwen2) and Gemma-3-4B-it(team2025gemma) for their efficiency and lightweight deployment potential. The mid-range segment (7–14B parameters) includes GLM-4V-9B(glm2024chatglm), InternVL3-9B(zhu2025internvl3), DeepSeek-VL-7B-chat(lu2024deepseek), LLaVA-v1.6-Mistral-7B-HF(li2024llava), MiniCPM-V-2.6-8B(yao2024minicpm), and Qwen2.5-VL-7B-Instruct(yang2024qwen2), which balance computational cost and multimodal reasoning capabilities through advanced attention mechanisms. At the large scale (19–32B parameters), we assess CogVLM2-Llama3-Chat-19B(hong2024cogvlm2), InternVL3-14B(zhu2025internvl3), and Qwen2.5-VL-32B-Instruct(yang2024qwen2), all of which leverage powerful visual encoders and extended context handling to tackle complex vision-language understanding tasks.

5.2. Experimental Setting and Resource

For all models, we set the temperature to $\tau=0.1$ and use top- $p$ sampling with $p=0.95$ for decoding. The maximum generation length is capped at 4096 tokens. For open-weight models, we adopt bfloat16 precision during inference. All experiments are conducted on A100 40G GPUs. The prompts used for each task are detailed in Appendix E.

5.3. Key Insights from Results

Tab. 4 reports the performance of the evaluated models on the Chart-to-Code task, whereas Tab. 5 presents their results on the controlled Chart-to-Table reconstruction task. Based on these results, we derive several key insights, with further discussion provided in the Appendix E.

1) GPT-4o demonstrates the best overall performance on the Chart-to-Code task, while Claude-3-7-Sonnet leads in the Controlled Chart-to-Table task. GPT-4o’s strength lies in its superior visual reasoning and structural consistency, yielding high pass rates and fidelity scores in code generation; meanwhile, Claude-3-7-Sonnet shows stronger table reconstruction capabilities, likely due to its alignment with structured data tasks—both clearly outperform open-source models.

2) Proprietary models lead in overall chart grounding, but open-source models are closing the gap in specific sub-tasks.GPT-4o achieves the highest overall score in chart-to-code generation (63.02) and strong F1 scores in chart-to-table reconstruction (F1-High: 37.72), outperforming all open models. However, Qwen2.5-VL-32B-Instruct surpasses GPT-4o in F1-High (41.04).

3) Model scale correlates with performance within model families, but architecture and training dominate cross-family comparisons.Overall scores increase with scale in Qwen2.5-VL and InternVL3 series(e.g., Qwen-VL: 3B–28.63, 7B–43.68, 32B–48.76; InternVL3: 2B–17.62, 9B–45.29, 14B–48.76). In contrast, CogVLM2-19B scores only 4.35 overall, suggesting architectural design and training alignment are more decisive than size alone.

4) Smaller models are competitive when aligned with task-specific objectives. Gemma-3-4B-it performs comparably to or better than some 7B+ models in layout accuracy and slight tolerance F1, showing that targeted training can offset limited scale, especially in lightweight deployment settings.

5) Chart-to-code generation reveals consistent weaknesses in color replication and data fidelity. Even top models (e.g., GPT-4o, InternVL3-14B) achieve lower scores in Color and Data Fidelity compared to structural elements like Title or Layout, indicating that fine-grained visual detail understanding remains a major bottleneck in generative chart reasoning.

6) Tolerance analysis highlights limitations in precise numerical reasoning. Across models, strict-F1 scores are substantially lower than those under slight or high tolerance, confirming that while macro-structure comprehension is relatively robust, exact data recovery—especially for decimals and axis-scaled values—remains unreliable.

7) Best Performance per Size. Among open-source models, Qwen2.5-VL-32B stands out as the top large model, coming close to GPT-4 in several areas. But for lower-budget scenarios, the 7B–14B tier models like InternVL3-9B and Qwen2.5-VL-7B offer strong performance at a fraction of the size. They both surpass 43 overall points, which is impressive relative to many other models under 10B. Meanwhile, certain 7B models not specifically tuned for this domain (like LLaVA 1.6 or DeepSeek) lag far behind. The comparison suggests that investing in domain-specific fine-tuning yields more benefit than simply adding parameters. A well-trained medium model can beat an untuned large model by a wide margin. This parameter efficiency perspective is crucial for real-world use, where deploying a smaller model that achieves, say, 80% of GPT-4’s performance could be far more practical.

8) The CLIP-based semantic alignment score tends to correlate positively with visual structure metrics like chart type and layout. For example, models with high layout alignment (GPT-4 at 80.7%, Claude-3-7-Sonnet at 76.5%) also show high CLIP scores (GPT-4 86.9, Claude-3-7-Sonnet 94.1) in Chart-to-Code. This suggests CLIP is a reliable proxy for detecting major structural errors—when models misidentify layout or chart type, CLIP similarity typically drops. While not sufficient for fine-grained correctness, CLIP complements structural metrics by capturing overall visual alignment.

5.4. Type Analysis

Fig. 5 presents the accuracy of the "data" attribute prediction across 30 chart types on the Chart-to-Code task, categorized into seven semantic groups in the context of the chart2code task. Among basic 2D Cartesian plots (e.g., bar, line, scatter), GPT-4o consistently outperforms other models, achieving over 0.8 accuracy in pie and areachart, while both Gemma3-4b and InternVL3-14b demonstrate stable but lower performance.

For polar plots such as barpolar and funnelarea, all models exhibit a marked performance drop, reflecting challenges in handling angular and radial encoding. In the 3D and high-dimensional category (scatter3d, surface, cone), accuracy further declines across the board, underscoring the difficulty of abstract spatial structures.

Matrix-style charts like heatmap and his2dcontour show modest improvements for GPT-4o, but still lag due to their dense grid-based semantics. In multidimensional projections (parcoords, scatterternary), prediction accuracy is near zero, suggesting current models lack sufficient capability to model complex mapping relationships.

Hierarchical and network-style charts (treemap, sunburst, sankey) reveal slight gains for GPT-4o, while others fail to generalize. Notably, in specialized financial plots (candlestick, ohlc, waterfall), GPT-4o again achieves high scores, indicating its better domain transfer ability. Overall, while GPT-4o demonstrates broad robustness, notable gaps remain in its ability to handle structurally complex or semantically specialized chart types, highlighting opportunities for future research.

6. Conclusion

In this work, we introduced ChartAnchor, a comprehensive benchmark for rigorous evaluating chart grounding in MLLMs. Unlike prior benchmarks that evaluate isolated tasks or focus on limited chart types, ChartAnchor provides a unified framework for both chart-to-code generation and controlled chart-to-table reconstruction, enabling holistic assessment of visual, structural, and numerical fidelity. It contains 8,068 chart–table–code triplets across 30 diverse chart types and two plotting libraries, capturing the complexity and heterogeneity of real-world visualizations. We further proposed a multi-level evaluation protocol that includes functional validity, visual consistency, data fidelity and perceptual similarity, and data accuracy. Empirical results across 14 leading MLLMs reveal that while state-of-the-art models such as GPT-4o demonstrate strong performance in structural parsing and layout reproduction, they still struggle with fine-grained data recovery and color fidelity. These findings underscore the pressing need for models that integrate symbolic reasoning and precise visual understanding. We hope ChartAnchor catalyzes future research on robust, semantically grounded chart comprehension, particularly in domains like science, finance, and policy where both interpretability and data integrity are paramount.

7. Limitations and Future Work

While the current version of ChartAnchor focuses on static chart understanding, we recognize that real-world visualizations increasingly involve interactive or dynamic elements—such as drill-down plots, animated data transitions, or multi-view dashboards. These charts introduce additional layers of complexity, including temporal structure, user interaction states, and evolving visual contexts.

In future work, ChartAnchor can be extended to incorporate interactive and time-varying visualizations. Specifically, benchmarks could be designed to simulate interaction flows and temporal data changes, enabling the evaluation of model capabilities in handling dynamic semantics and multi-state rendering. Such an extension would further broaden the applicability of ChartAnchor to real-world analytical environments.

Appendix

Appendix A Detailed Data Collection and Processing Procedures

In this appendix, we provide a comprehensive description of our data collection, code synthesis, and quality filtering procedures for constructing the ChartAnchor dataset, a large-scale corpus of chart–table–code triples designed to support chart understanding and structured generation tasks. The goal of our pipeline is to ensure semantic fidelity, visual diversity, and execution reliability, making the dataset suitable for rigorous evaluation of large multimodal models (MLLMs).

A.1. Data Sources and Corpus Composition

We collect a total of 230,549 raw chart–table pairs, which serve as the foundation for constructing the ChartAnchor corpus. These samples are derived from two sources:

•

218,549 chart–table–code triplets crawled from open-source visualization platforms, where each sample includes a rendered chart image, the structured data table, and the corresponding plotting script (mainly written in Plotly).
•

12,000 chart–table pairs sampled from four existing chart-realted datasets that provide image–table pairs but do not include source code.

In the remainder of this section, we describe the composition and characteristics of both sources in detail.

1. Existing Datasets. To bootstrap the construction of ChartAnchor, we leverage four well-known chart-centric datasets: PlotQA, DVQA, FigureQA, and VisText. These datasets collectively comprise over 750,000 chart samples, each offering paired chart images and tabular data, but lacking executable plotting code. From each dataset, we uniformly sample 3,000 representative chart–table pairs, yielding a total of 12,000 initial samples for downstream code synthesis. The sampled charts span five major types: bar, line, scatter, area, and pie charts.

Below, we briefly describe the origin and structure of each dataset:

•

PlotQA is a large-scale dataset constructed from real-world sources such as the World Bank, government portals, and open data platforms. It covers diverse domains including economics, health, education, and the environment. It contains 224,377 chart images, primarily bar, line, and scatter plots, each paired with structured table data and metadata.
•

DVQA is a synthetic dataset focused on bar chart understanding, comprising 300,000 high-resolution images generated via Matplotlib. Each sample is associated with its underlying data table and metadata describing chart layout elements.
•

FigureQA contains over 100,000 synthetic chart images generated using the Bokeh library, covering five figure types with structural annotations such as bounding boxes and labels.
•

VisText comprises 12,441 chart–caption samples, each including a chart image, data table, and scene graph. Charts are rendered using Vega-Lite with real-world data from Statista.

Type	Scatterpolar	Cone	Line3d	Carpet	Barpolar	Mesh3d	Ohlc	Line	Histogram2d
Num	8193	384	10000	164	1599	10000	3949	10000	4873
Type	Areachart	Box	Scatterternary	Waterfall	Heatmap	Scatter	Scatter3d	Surface	Histogram
Num	5740	9530	3915	395	9980	10000	7300	10000	10000
Type	Treemap	Violin	Parcoords	Funnelarea	Funnel	Sankey	Candlestick	Contour	Sunburst
Num	1122	7960	3567	219	716	8910	4750	10000	2962
Type	bar	pie	histogram2dcontour	Density Tile Map	Tile Map	Atlas Map	Choropleth Atlas Map	Choropleth Tile Map	Image-based Table
Num	8920	10000	5250	585	9382	9675	9260	513	8736

Table 6. Number of samples per chart type in the newly curated dataset.

2. Newly Curated Dataset. To enhance chart diversity and obtain source code supervision, we crawl 218,549 chart–table–code triplets from open-source visualization communities. These samples are created by users across a wide range of domains and include full plotting scripts. Each sample contains a rasterized chart image, the underlying data table, and a Python script that can regenerate the visualization.

This collection spans over 36 chart types, including both standard (e.g., bar, line, scatter) and specialized forms (e.g., candlestick, violin, sankey). Table 6 summarizes the sample distribution by chart type.

A.2. Detailed Code Generation and Augmentation for Existing Datasets

To address the lack of source code in existing chart datasets, we design a parameterized code generation pipeline that translates chart metadata into executable Python plotting scripts. This enables us to construct chart–table–code triplets from datasets that originally only contain chart images and tabular data. The pipeline consists of three sequential stages: (1) semantic mapping, (2) visual attribute parameterization, and (3) controlled data augmentation.

Semantic Mapping.

We first parse the metadata associated with each chart (e.g., chart type, data structure, labels, axis information) and map it to corresponding primitives in popular visualization libraries. For example, bar charts are mapped to calls like plt.bar(), line charts to plt.plot(), and scatter plots to plt.scatter(). We also infer high-level layout logic such as multiple series plotting, stacked vs. grouped bar configurations.

Visual Attribute Parameterization.

We define a structured set of visual attributes that govern the appearance of chart renderings. These attributes cover visual elements such as colors, strokes, fonts, axes, legends, and layout. Each attribute corresponds to a configurable parameter in the plotting code and forms the basis for subsequent augmentation.

•

Color schemes: parameters defining the color of key visual elements including lines, bars, markers, and background.
•

Stroke and marker styles: properties such as line width, dash pattern, and marker shape applicable to strokes or data points.
•

Font settings: parameters specifying font size, family, and weight for chart titles, axis labels, and tick labels.
•

Axis configuration: includes axis visibility, tick mark density and orientation, label formatting, and scaling behavior (e.g., linear vs. log).
•

Legend configuration: layout options including visibility, location, frame style, and padding.
•

Canvas layout: overall figure width, height, and aspect ratio, affecting spatial organization and density.

Systematic Augmentation.

To further increase style diversity, we apply controlled random perturbations over the sampled parameters. All augmentations are range-constrained to preserve semantic structure while introducing sufficient variability. The applied strategies include:

•

Color perturbation: visual element colors are augmented through multi-level sampling and transformation. Initial base colors are perturbed in HSV color space by applying random shifts to hue ( $\pm$ 0.2), saturation ( $\pm$ 0.25), and brightness ( $\pm$ 0.25), resulting in perceptually similar yet distinct styles. For elements requiring visual separation (e.g., gridlines or multiple series), two contrasting strategies are adopted: (i) same-family perturbation with minimal hue deviation to maintain stylistic consistency, and (ii) complementary sampling, where colors are rotated approximately 180 degrees in hue and slightly jittered to avoid exact symmetry. All foreground colors—such as text, ticks, and gridlines—are dynamically adjusted based on the chart background to ensure a minimum contrast ratio of 3:1, as computed using relative luminance. If automatic contrast resolution fails, fallback binary colors (black or white) are applied to maintain legibility.
•

Line and stroke styling: visual stroke properties—including line width, line pattern, and marker shape—are randomly selected from constrained sets. Line widths are sampled uniformly from a predefined range (e.g., 1.2 to 3.5 pt), and styles are chosen from standard patterns such as solid, dashed, dotted, and dash-dot. This variation simulates a broad range of visual densities and chart semantics, while ensuring clarity and readability.
•

Grid and frame styling: the presence of gridlines is toggled with a fixed probability (e.g., 70%). If enabled, grid properties including color, alpha transparency (range: 0.2–0.6), and line style are randomly assigned. Grid color may be either a low-contrast perturbation of the primary visual element (same-family) or a dynamically selected complementary color that satisfies a minimum contrast ratio relative to the background. Chart frame (spine) color and width are also independently perturbed if specified, or otherwise randomly assigned.
•

Tick and axis label formatting: font size for tick labels is sampled from a narrow range (e.g., 8–12 pt), and axis label font size is sampled separately (e.g., 10–14 pt). Tick direction, length, and visibility are randomized across axes. Label rotation angles are applied conditionally when present, or left unset to trigger automatic formatting. Axis spine visibility and style may also vary with domain-specific settings.
•

Text and title layout: title font size is sampled within a broader range (e.g., 12–18 pt) to accommodate both compact and expanded figure layouts. To preserve clarity in narrow plots, chart titles are automatically wrapped at fixed character widths (e.g., 50 characters per line) to avoid horizontal overflow and truncation, ensuring consistent rendering across canvas aspect ratios.
•

Legend configuration: legend display is toggled probabilistically. When shown, layout options including position (e.g., top-right, bottom-center), spacing, and frame style are randomly selected. This simulates visual clutter or compression effects common in real-world plots with many categories.
•

Canvas and layout variation: figure width and height are sampled from uniform distributions (e.g., width = 6 ± 3 units, height = 5 ± 2 units), producing aspect ratios ranging from portrait to landscape. These adjustments impact element scaling, whitespace, and overall plot density, thereby exposing models to varying layout constraints and visual balance conditions.

These attribute-level augmentations collectively simulate a broad range of real-world chart styles. By systematically varying color schemes, marker designs, axis configurations, legend layouts, font properties, and layout structures, the resulting code-image pairs exhibit substantial stylistic diversity while preserving semantic fidelity. This enables more robust training and evaluation of models in chart understanding and generation tasks. All scripts are programmatically verified for syntax correctness and rendering completeness, ensuring reproducibility and consistency across the constructed dataset.

A.3. Filtering Strategy and Statistics

Filtering Stage	Removed	Retained
Completeness Check	51,582	178,967
Structural Filtering	52,963	126,004
Deduplication	62,525	63,479
Executability Check	15,011	48,468
Manual Filtering	40,400	8,068

Table 7. Sample counts at each filtering stage.

Automated filtering.

– Completeness filtering: We removed all samples missing any of the core components: structured table, chart image, or generation code. This includes examples with null or corrupt image files, empty tables, or scripts lacking plotting calls. A total of 51,582 examples were eliminated at this step.

– Structural filtering: We excluded chart instances whose structure could not be reliably reconstructed into code without external visual assets. This includes geospatial plots (e.g., choropleth tile map), rasterized or image-based tables, and charts containing embedded logos and background images that are not specified in the underlying table. In addition, we filtered out samples whose generated code exceeded a predefined length threshold, indicating excessive verbosity, redundant operations, or inclusion of unrelated plotting logic. This step removed 52,963 samples in total.

– Deduplication: For each data table, we extracted column-level features including column type (categorical or quantitative), column length, and a representative statistic determined by type: the most frequent value for categorical columns and the mean for quantitative columns. These features were computed for all columns and concatenated in column order to form a string-based table signature. Charts with identical signatures were treated as duplicates, and only one instance was retained. This filtering step removed 62,525 structurally duplicated samples.

Example.

The following table:

City	Category	Score
Paris	A	88.0
Paris	B	92.0
London	A	84.0
London	B	94.0

is processed column by column to compute the table signature:

•
Column 1 (City):
- –
  
  Type: Categorical
- –
  
  Length: 4
- –
  
  Representative statistic: Most frequent value = Paris
•
Column 2 (Category):
- –
  
  Type: Categorical
- –
  
  Length: 4
- –
  
  Representative statistic: Most frequent value = A
•
Column 3 (Score):
- –
  
  Type: Quantitative
- –
  
  Length: 4
- –
  
  Representative statistic: Mean = 89.5

These features are concatenated in column order to produce the table signature:

categorical4Pariscategorical4Aquantitative489.5

If another chart shares the same signature, it is considered structurally equivalent and is filtered out.

– Executability check: Each Python script was executed in an isolated environment. We discarded any sample whose script resulted in errors (e.g., due to missing fields or malformed syntax), produced no visual output, or generated a blank or invalid image file. This step filtered 15,011 additional cases.

After automated filtering, a total of 48,468 high-confidence samples were retained from the crawled corpus.

Dimension	Accept	Borderline	Reject
Semantic Accuracy	All columns in the table are correctly encoded in the chart; axis titles match column names; all data series are included; data-to-visual mappings are accurate and complete.	Most relevant columns are included; minor mismatches in field-to-axis mapping, partial omission of non-critical fields, or inaccurate axis labeling may exist.	One or more key columns are missing or misused; axis assignments do not match table structure; values are incorrectly encoded or hardcoded; chart misrepresents the data.
Visual Clarity	All text is readable; tick marks, labels, and gridlines are well-aligned; spacing and font sizes are appropriate; the chart has no overlaps or clipping.	The chart is generally legible but contains minor issues such as crowded labels, small text, or slight misalignment of elements.	The chart contains severe layout problems, including overlapping text, unreadable labels, distorted scaling, or clipped elements that obstruct interpretation.
Stylistic Diversity	The chart uses varied formatting choices in color, font size, spacing, or layout; visual elements (e.g., legend placement, label orientation) differ from other charts of the same type.	Some formatting differences are present, but the chart closely resembles many others in the same category; variation is minimal.	Formatting is nearly identical to multiple other charts; visual parameters (e.g., spacing, label orientation, font, and color) are reused without change.

Table 8. Review rubric for chart–table–code triples.

Human Filtering Protocol.

Each chart–table–code triple was manually reviewed to ensure semantic correctness, visual interpretability, and style variation. Reviewers were graduate-level annotators with experience in Python scripting and data visualization. A total of six reviewers participated in the process, which was completed over the course of ten days.

The review was conducted in two passes: independent blind annotation followed by adjudication in case of disagreement. To ensure intra-group consistency, samples were first grouped by structural chart type (e.g., bar, line, scatter). Each triple was then evaluated along three axes—semantic accuracy, visual clarity, and stylistic diversity—following the rubric in Table 8.

Each dimension was assigned one of three labels: (1)Accept, (2)Borderline, or (3)Reject. Samples receiving a double-Reject on any axis were removed. Disagreements or borderline cases were reviewed by a third annotator, who was allowed to execute or minimally adjust code to resolve ambiguity.

In total, 48,468 samples were flagged for human review, of which 40,400 were removed, 8,068 accepted without change.

Appendix B More Analysis about Dataset

We tokenize each code script in the benchmark and compute the token length for each example. The analysis shows an average length of approximately 628 tokens with a standard deviation of 466, and a minimum of 58 tokens. This indicates that the dataset includes both concise, logically clear scripts and longer, more complex ones, reflecting a broad range of task difficulty and coverage of real-world scenarios. Such diversity provides a solid foundation for evaluating the generalization capabilities of multimodal models across varying levels of complexity.

We categorize all table columns in the dataset into three types: string, numeric, and date. Numeric columns include both integers and floating-point numbers. Columns that do not meet either criterion are classified as strings by default. Our analysis shows that 71% of the columns are numeric, 23% are string, and 6% are date. This distribution indicates that numeric fields dominate the dataset, aligning with the inherently quantitative nature of most data visualizations. At the same time, the substantial presence of string and date fields highlights the dataset’s semantic diversity, supporting categorical labeling and temporal trends. These findings demonstrate that ChartAnchor offers broad coverage of semantic structures commonly found in real-world data analysis tasks.

Appendix C Broader Impacts

This work introduces a benchmark designed to evaluate the chart grounding ability of multimodal models through structured generation tasks—specifically, producing executable code and aligned tabular data from visual input. By formulating chart-centric understanding as a code- and table-grounded task, the benchmark enables more precise assessment of a model’s capacity to recover structured semantics from complex visualizations. The dataset is constructed from publicly available and license-compliant sources, with a focus on semantic traceability, syntactic validity, and reproducibility.

To mitigate potential risks, we adopt several safeguards: (1) the dataset emphasizes structured and verifiable content; (2) our filtering and annotation protocols (see Appendix A) enforce consistency across modalities; (3) the evaluation suite includes tests for code execution, alignment fidelity, and structural coverage.

While the benchmark is designed to support research in grounded and interpretable generation, we acknowledge the possibility of unintended use. For instance, models trained or evaluated on ChartAnchor might be applied in automated settings without verification, potentially leading to misleading outputs. Although the benchmark itself does not directly enable such misuse, we recommend that future applications incorporate human oversight, validation mechanisms, and appropriate deployment constraints. Ensuring output traceability is particularly important when models are used in domains such as scientific computing, data journalism, and business reporting.

We hope that ChartAnchor serves as a resource for advancing multimodal systems that prioritize structured reasoning, factual alignment, and transparency.

Appendix D Model License

Table 9 summarizes the licenses of all models evaluated in ChartAnchor, including both model weights and accompanying code repositories.

Model	Model License	Code License
GPT-4o	Proprietary	Proprietary
Claude-3-7-Sonnet	Proprietary	Proprietary
InternVL3-2B	Apache 2.0	MIT
Qwen2.5-VL-3B-Instruct	Apache 2.0	Apache 2.0
Gemma-3-4B-it	gemma	Not Applicable
DeepSeek-VL-7B	deepseek	MIT
LLaVA-v1.6-Mistral-7B	Apache 2.0	Apache 2.0
Qwen2.5-VL-7B-Instruct	Apache 2.0	Apache 2.0
MiniCPM-V-2.6-8B	minicpm	Apache 2.0
InternVL3-9B	Apache 2.0	MIT
GLM-4V-9B	glm-4	Not Applicable
CogVLM2-Llama3-Chat-19B	llama3 + cogvlm2	Apache 2.0
InternVL3-14B	Apache 2.0	MIT
Qwen2.5-VL-32B-Instruct	Apache 2.0	Apache 2.0

Table 9. Summary of licenses in models that are evaluated in ChartAnchor. Entries marked with “Not Applicable” indicate that authors do not have an explicit code license displayed within the codebase or model checkpoint page.

Appendix E Experiments

E.1. Prompts

We design tailored input prompts for each task to guide model behavior effectively. Figure 6 illustrates the prompt used in the Chart-to-Code task, which specifies the desired output format and allows the model to choose between visualization libraries (e.g., Plotly or Matplotlib). Figure 7 further presents the prompt format for the Controlled Chart-to-Table task, where key placeholders are dynamically adjusted based on the provided table headers.

E.2. Additional Analysis

As shown in Table 4 and Table 5, we can see:

- Color Feature Presents Unique Challenges in Visual Decoding. Color accuracy remains the lowest-scoring aspect within the Visual Structure Consistency metrics, even for top-performing models such as GPT-4o (34.64), Claude-3-7-Sonnet(40.76) and InternVL3-14B (26.13). This suggests that fine-grained color differentiation poses unique challenges for current visual encoders, especially in complex or low-contrast chart regions. In many cases, the visual abstraction processes used by these models may reduce sensitivity to precise pixel-level color features.

- Claude-3-7-Sonnet achieves the highest CLIP Score despite lower structural fidelity. Claude-3-7-Sonnet obtains the highest Clip Score (94.11), surpassing GPT-4o (86.88), indicating strong perceptual alignment with original chart styles. However, its overall Chart-to-Code accuracy (61.18) and structural metrics (e.g., Color: 40.79) remain lower, revealing a gap between visual realism and semantic correctness. This suggests that high CLIP similarity does not imply accurate structural grounding.

- High Pass Rate but Low Fidelity: Another failure mode is models generating syntactically valid outputs that “mask critical failures” in content. For example, DeepSeek-VL-7B successfully produces parsable code nearly every time ( about 98% pass rate), yet its data extraction is almost nonexistent (strict F1 ¡1%). It often writes basic chart code that runs but does not capture the actual data or visual details. This strategy yields a high functional score but extremely low data fidelity, indicating the model is defaulting to trivial or placeholder outputs to avoid errors rather than truly understanding the chart.

- A high CLIP score does not guarantee accurate data reconstruction. A model may generate a chart that appears visually correct—with the right chart type, structure, and even similar color distribution—while the underlying data is incorrect. For example, Claude-3-7-Sonnet achieves a high CLIP score (indicating strong visual similarity), but its data fidelity (e.g., F1) is only comparable to GPT-4 and slightly below Qwen2.5-VL-32B, which has the best high-tolerance F1 (39.6) but a much lower CLIP score (64.04). This shows that visual similarity and data accuracy can diverge. As noted in the introduction, a model may “reproduce a chart’s appearance while silently altering the data.” Thus, CLIP and data F1 are complementary: the former captures visual errors, the latter semantic ones. Both are essential for evaluating chart fidelity.

E.3. Analysis Across Chart Types

Figure 8 shows that overall model performance varies significantly by chart type. Models perform best on simple 2D Cartesian charts like bar and line, which have consistent structures and are likely well-represented in training data. In contrast, performance drops sharply on 3D, matrix-style, and hierarchical charts, which involve complex layouts or dense data encoding. Financial charts show moderate stability, likely due to their standardized formats.

Compared to data fidelity prediction (Figure 5), some differences emerge. For instance, pie charts score high in data recovery but only moderately in overall performance, suggesting structural elements (e.g., legends) are hard to reproduce. Meanwhile, 3D and matrix charts perform worse in data accuracy than in overall score, highlighting the difficulty of recovering exact values from visually dense charts.

Figure 9 shows model performance on the chart-to-table task under slight tolerance. Basic Cartesian charts (e.g., bar, line) achieve the highest F1 scores due to their clear value mappings. In contrast, 3D, polar, and matrix-style charts perform poorly across models, reflecting challenges from visual distortion or dense layouts. Notably, hierarchical charts (e.g., sunburst, treemap) perform better here than in data attribute prediction (Figure 5), suggesting that structured table formats help guide value extraction despite visual complexity.

Appendix F Case Study of ChartAnchor

Example 1: Figure 10 illustrates the output of a Chart-to-Code task, comparing the gold reference image with the charts generated by three models: GPT-4o, Gemma3-4b, and InternVL3-14b. Among them, GPT-4o demonstrates the highest fidelity in visual styling, closely mimicking the original chart’s color fill, line smoothness, and overall aesthetic. In contrast, InternVL3-14b more accurately captures the data trends and year-to-year fluctuations, reflecting the original curve’s structure with greater numerical precision.

Example 2: As shown in figure 11, GPT-4o offers the most faithful reproduction overall, closely matching the original chart’s data values and bar patterns, including the distinctive crosshatch fill—though it uses a dark background instead of light. In contrast, both Gemma3-4b and InternVL3-14b lack the patterned bars and display fewer, less precise data points, resulting in lower fidelity in both appearance and accuracy. While all models preserve the general upward trend, GPT-4o stands out for its high consistency in both visual styling and numerical detail.

Example 3: As shown in figure 12, in this example, GPT-4o is the only model that successfully replicates the candlestick chart structure from the gold image, preserving both the visual format and financial data elements. InternVL3-14b fails to produce a candlestick chart, instead outputting a misleading bar chart with incorrect trends. Gemma3-4b completely fails to render a valid image, as indicated by the gray placeholder.

Example 4: As shown in figure 13, in this example, Claude-3-7-Sonnet is the closest to replicating the contour style of the gold image, correctly preserving the shape and gradient regions, though with reduced detail. GPT-4o fails to generate contours, instead producing a simplified heatmap with horizontal bands that ignore spatial gradients. InternVL3-14b fails entirely, indicated by the gray placeholder.

Example 5: As shown in figure 14, only Claude-3-7-Sonnet successfully generates valid funnelarea charts, although the layout differs from the original: rearranges their positions. GPT-4o and InternVL3-14b both fail to render any output, shown as gray placeholders. While Claude’s version lacks stylistic fidelity, it maintains correct data values and category separation.

Example 6: As shown in figure 15, GPT-4o is the only model that successfully reproduces the heatmap format of the gold image, including the layout of cities and categories. While the color mapping is slightly off, the structural format is well preserved. Gemma3-4b and InternVL3-14b both deviate significantly by converting the data into grouped bar charts. Although their bar heights roughly reflect the underlying values, they lose the compact matrix layout and visual impact of the original.

Example 7: As shown in figure 16, all models correctly reproduce a pie chart with the same numerical values as the gold image. GPT-4o and InternVL3-14b maintain both the correct labels and proportions, although GPT-4o’s layout is elliptical rather than circular. Gemma3-4b also retains the right proportions but uses a completely different color scheme and label placement. Overall, GPT-4o and InternVL3-14b achieve high fidelity, while Gemma3-4b is accurate in data but less faithful in appearance.

Example 8: From figure 17, we can see, InternVL3-14b is the closest to the gold image, successfully generating both the box-and-whisker plot and the dot plot with a similar layout, color scheme, and annotated sample. GPT-4o captures only the dot plot and omits the box plot entirely, resulting in partial fidelity. Gemma3-4b fails to render any output, as shown by the gray placeholder. While InternVL3-14b slightly alters the orientation and spacing, it delivers the most accurate visual and structural reproduction overall.

Example 9: From the gold table (Table 10) and the generated tables (Figure 18), by comparing the row structures and numeric alignments, we observe that claude-3-7-Sonnet produces a table closely aligned with the gold table, with all values either correct or slightly off. Qwen2.5-VL-7B shows noticeable deviations in the ”RDS 18–49” and ”RDS Total” columns for 2014 and 2016. Qwen2.5-VL-32B gets closer but still introduces uniform values in 2015 that deviate from the gold data. Claude-3-7-Sonnet demonstrates the highest data fidelity and variation consistency across rows.

Year	RDS 18_49	RDS Total	TSN 18_49	TSN Total
2014	58,000	191,000	209,000	660,000
2015	63,000	201,000	157,000	535,000
2016	41,000	142,000	170,000	553,000

Table 10. Example 9: Gold Table

theta	Today-r	Mean-r
AL29	66.87	44.09
AL30	76.31	40.65
AE38	26.49	39.08
AL41	43.25	50.65
AL35	40.96	45.12
GD29	56.15	44.77
GD30	51.42	43.54
GD38	6.06	46.96
GD41	25.54	51.53
GD35	26.17	41.69

Table 11. Example 10: Gold Table

Example 10: From the gold table (Table 11) and the generated tables (Figure 19), we can see, in this example, Claude-3-7-Sonnet is the only model that correctly identifies the categorical theta labels (e.g., AL29, GD35) and extracts reasonable numerical values for both ”Today-r” and ”Mean-r,” achieving the highest alignment with the gold table. In contrast, Qwen2.5-VL-7B and Qwen2.5-VL-32B fail to capture the label names and instead generate numerical theta angles, indicating a misunderstanding of the scatterpolar chart structure. Their values also significantly diverge from the reference data, limiting their usability for structured analysis.

Example 11: From the gold table (Table 12) and the generated tables (Figure 20), we can see, Claude-3-7-Sonnet is the only model that accurately captures both the structure and values of the gold table. It correctly identifies incremental changes (including positive and negative values) and matches the final ”Price current” result. In contrast, Qwen2.5-VL-7B and Qwen2.5-VL-32B misinterpret the chart as cumulative values rather than stepwise deltas. As a result, their tables lose the core logic of a waterfall breakdown and deviate significantly from the reference.

trace0-x	trace0-y
Price previous year	200,000.0
Quantity difference	-10,000.0
Currency impact	-10,000.0
Market impact	15,000.0
Price reduction	-10,000.0
Not controlled	-25,000.0
Price current	100,000.0

Table 12. Example 11: Gold Table

Appendix G Chart Examples

This section presents visual examples of the chart categories included in our benchmark. Each image illustrates a distinct chart type, showcasing the diversity in structure, data encoding, and visual design. Figures 21 display representative samples from all 30 categories.

Statistics	Value
Chart Images
Total Charts	8,068
Avg. Width	3,346
Avg. Height	2,266
Aspect Ratio	1.43
Brightness Std	42.35
Entropy	1.41
Python Code
Mean Chars	1,469
Mean Tokens	627.67
Table
Mean Rows	20.35
Mean Cols	3.05

ChartAnchor: Chart Grounding with Structural-Semantic Fidelity and Data Recovery

Abstract.

1. Introduction

2. Related Work

3. The ChartAnchor Benchmark

3.1. Task Definitions

3.2. Data Collection

3.2.1. Data Source

3.2.2. Code Generation and Augmentation for Existing Data- sets.

3.2.3. Filtering.

3.3. Dataset Analysis

3.4. Comparition with other benchmarks

4. Evaluation Metrics

Semantic Data Fidelity.

5. Evaluation

5.1. Benchmarking Models

5.2. Experimental Setting and Resource

5.3. Key Insights from Results

5.4. Type Analysis

6. Conclusion

7. Limitations and Future Work

Appendix

Appendix A Detailed Data Collection and Processing Procedures

A.1. Data Sources and Corpus Composition

A.2. Detailed Code Generation and Augmentation for Existing Datasets

Semantic Mapping.

Visual Attribute Parameterization.

Systematic Augmentation.

A.3. Filtering Strategy and Statistics

Example.

Appendix B More Analysis about Dataset

Appendix C Broader Impacts

Appendix D Model License

Appendix E Experiments

E.1. Prompts

E.2. Additional Analysis

E.3. Analysis Across Chart Types

Appendix F Case Study of ChartAnchor

Appendix G Chart Examples

3.2.2. Code Generation and Augmentation for Existing Data-
sets.