Fernandosr85/nl2vis-chart-spec-adapter

NL2VIS Chart Spec Adapter

LoRA adapter for converting natural-language chart requests into visualization specifications, fine-tuned on Llama-4-Scout-17B-16E-Instruct (109B parameters across 16 experts, 17B active per token) via Adaption's AutoScientist platform.

Given a database schema and a request in plain English, the adapter emits one line with five fields:

CHART_TYPE || X_COLUMN || Y_COLUMN || GROUP_COLUMN || AGGREGATE

---

Task

Closed-vocabulary specification, not open generation. Every field comes from a vocabulary external to the model:

Field	Vocabulary	Source
CHART_TYPE	BAR, LINE, SCATTER, PIE	nvBench
X_COLUMN	columns of the schema in the prompt	Spider tables.json
Y_COLUMN	columns, or * for a plain row count	Spider tables.json
GROUP_COLUMN	columns, or NONE	Spider tables.json
AGGREGATE	NONE, COUNT, SUM, AVG, MIN, MAX	SQL grammar

Because the column vocabulary is the schema supplied in the prompt, a predicted column outside it is a checkable hallucination — a set lookup, not a judgement call. That splits column errors in two:

COLUMN_HALLUCINATION   the column does not exist in the schema
COLUMN_SCOPE_ERROR     the column exists but sits on the wrong channel

Measured on the holdout, two frontier models produced 0.0% hallucination: the output stays inside the schema, and every column error is a scope error.

---

Evaluation results

Training win rates (Adaption internal)

Model	Win Rate (in-domain)
Base (Llama-4-Scout-17B-16E-Instruct)	2%
Adapted (adaption_nl_to_viz_spec_pairs)	98%

A +96 point improvement.

Cross-domain, without forgetting. On the Data-analysis-visualization reference category the win rate rises 45 → 56. The adapter improved outside its own dataset rather than trading general capability for in-domain accuracy — worth noting because task-specific adapters usually pay that price.

How to read a base score of 2

Two frontier models were measured on this exact task at 45.7% and 34.7% full exact-match. A base score of 2 therefore does not mean the model cannot reason about visualization. It means it does not emit a parseable five-field spec: win rate compares responses, and prose describing a chart loses to a specification every time.

A large share of those 96 points is format adherence rather than visualization competence. That is precisely what makes the adapter usable in a rendering pipeline, but the honest headline is "the base model does not emit a parseable spec; the adapter does, and gets it right" — not "the base model cannot do visualization".

Zero-shot baselines on the published holdout

300 rows, text-only, two scoring regimes. Strict counts COUNT(<column>) against COUNT(*) as an error; equivalence-aware does not, because they render the identical chart.

Method	Strict full EM	Equivalence-aware
Majority (BAR/NONE/COUNT)	0.0%	—
Lexical (best name-overlap column)	9.3%	—
Frontier model A	45.7%	67.0%
Frontier model B	34.7%	68.0%
This adapter	[not measured]	[not measured]

The lexical baseline is the control that matters: it picks the schema column whose name best overlaps the request words, and always answers BAR / NONE. A model that cannot clearly beat it is doing string matching rather than visualization grounding. The margin was +36.3 points.

---

Two measurement artifacts found in nvBench

1. COUNT(<column>) and COUNT(*) are the same chart.

38.7% of the corpus has AGGREGATE=COUNT with a named column as gold y_column. And:

100% − 38.7%                          = 61.3%
y_column accuracy, model A (measured) = 61.3%
y_column accuracy, model B (measured) = 61.7%

Not a coincidence. Both models got y_column right everywhere except where gold writes COUNT(paperID) and they answer COUNT(*). Scoring leniently under COUNT moves y_column from 61.3% to 96.3%, and full exact-match from 34.7% to 67.0%.

2. vis_obj.classify is not the grouping column.

It holds the distinct values of that column — ['F','M'] for GROUP BY Sex, Rank. The name must be parsed from the SQL GROUP BY clause, taking the term that is not the x column. A parser reading classify as a column name produces silently wrong gold on the 7.7% of records that are grouped charts.

Related: AGGREGATE parsed from y_name disagrees with the SQL on 3.4% of records. The SQL is the specification; y_name is a display string.

---

Two fields cannot carry a claim

Field	Majority value	Share
CHART_TYPE	bar	81.2%
GROUP_COLUMN	NONE	92.3%

Always answering bar scores 81.2%; always answering NONE scores 92.3%. A frontier model reached 89.7% on group_column — 0.7 points above the majority baseline, i.e. no signal at all.

The headroom sits in AGGREGATE (84.3% against a 54.2% majority) and x_column (92.7%), and the training mixture is stratified accordingly.

---

The hypothesis that did not survive

This work was built to test whether the failure mode is column attribution: when a schema offers confusable columns (sales_amount/sales_count, region/sub_region), does the model attach the right one to the right channel?

A confusable slice covering 50.3% of records was defined in advance, with a 10-point threshold. The measured gap was 3.2 points. The models do degrade on confusable schemas, but not much, and the headroom is not concentrated where the hypothesis predicted.

It is stated here because a diagnostic that reports only its confirmations is not a diagnostic.

---

Model details

Field	Value
Base model	meta-llama/Llama-4-Scout-17B-16E-Instruct (109B total, 17B active per token, 16 experts)
Trained model name	adaption_nl_to_viz_spec_pairs
Training method	Supervised Fine-Tuning (SFT) + LoRA
LoRA rank (r)	64
LoRA alpha	128
LoRA dropout	0
Trainable modules	attention (q/k/v/o_proj) plus expert FFN (feed_forward.*) and shared-expert FFN (shared_expert.*)
Epochs	5
Learning rate	1e-4 (cosine, warmup 0.05)
Weight decay	0.03
Max grad norm	1.0
Train on inputs	false
Data format	chat
Adapter size	829 MB

On the module selection. Unlike an attention-only adapter, this configuration reaches the mixture-of-experts feed-forward layers — both the routed experts and the shared expert. On a 109B-parameter MoE base that is where most of the parametric knowledge sits, and it accounts for the 829 MB footprint against the ~200 MB of a comparable attention-only LoRA.

On the model name. The 17B in Llama-4-Scout-17B-16E is the active parameter count per token, not the model size: the base carries 109B parameters across 16 experts and routes a subset per token. Inference needs VRAM for the full footprint — sparse activation reduces compute, not memory.

Inference note. data_format was chat and train_on_inputs was false. Wrap the request as a user turn via the tokenizer's chat template; raw text will not match the training format.

Archive note. The distributed archive may be zstd-compressed despite a .tgz extension. If tar -xzf fails, use tar --use-compress-program=unzstd -xf.

---

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base)
base_model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.float16, device_map="auto",
)
model = PeftModel.from_pretrained(
    base_model, "Fernandosr85/nl2vis-chart-spec-adapter",
)

# The field definitions are part of the trained prompt. GROUP_COLUMN in
# particular must be spelled out: an earlier prompt said only "a schema column
# or NONE", and the model answered with the x column again, scoring 72.0%
# against an 89.0% majority baseline -- worse than always answering NONE.
prompt = """You are converting a natural-language request into a visualization specification.

DATABASE SCHEMA:
- Region_name [text]
- Country [text]
- Population [number]
- Area_km2 [number]

REQUEST:
Show the average population by country as a bar chart.

Answer with exactly one line in this format, nothing else:
CHART_TYPE || X_COLUMN || Y_COLUMN || GROUP_COLUMN || AGGREGATE

CHART_TYPE: one of BAR, LINE, SCATTER, PIE.

X_COLUMN: the schema column plotted on the horizontal axis, copied verbatim.

Y_COLUMN: the schema column the aggregate is computed over, copied verbatim.
Use * ONLY when the request asks for a plain count of rows with no column named.
If the request counts or sums a specific column, name that column instead.

GROUP_COLUMN: the extra series dimension that splits the data into multiple
series, as in a grouped or stacked chart. It MUST be different from X_COLUMN.
Answer NONE for an ordinary single-series chart. Most charts are single-series,
so NONE is the common answer. Do not repeat X_COLUMN here.

AGGREGATE: one of NONE, COUNT, SUM, AVG, MIN, MAX."""

messages = [{"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt",
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

# Expected:
# BAR || Country || Population || NONE || AVG
#
# Validate before rendering: every column in the output should exist in the
# schema you supplied. That is a set lookup, and it is worth running.

do_sample=False on purpose — specification is deterministic, and you want the most likely label rather than a sample.

---

Training dataset

Fernandosr85/adaption-nl-to-viz-spec-pairs

13,635 prompt/completion pairs derived from nvBench (7,247 VIS records, 25,762 NL queries, 152 databases), with schemas resolved against Spider — all 152 matched, 0 missing. nvBench labels were validated by 23 experts and 300+ crowd workers.

Step	Detail
Label source	nvBench, external to this project
Gold parsing	AGGREGATE from the SQL, not y_name (3.4% disagreement)
Group column	From the SQL GROUP BY, the term that is not x
Split	By VIS key; no eval key in the training set (asserted in code)
NL per record	2 of the up-to-9 paraphrases nvBench ships
Stratification	By AGGREGATE, the field with measurable headroom
Rare-class boost	MIN and MAX 3× from their 1.3% and 1.5% natural share
Completions	Kept verbatim, not platform-regenerated

On the COUNT normalisation. Under COUNT the y column carries no visual information, so all COUNT cases were normalised to *. Training on the gold as-is would teach the adapter to guess which arbitrary column the annotator happened to write.

On what the mixture bets on. CHART_TYPE and Y_COLUMN are near-solved by frontier models (99.0% and 96.3%) and are included only to keep the output format intact. GROUP_COLUMN has no signal. The bet is AGGREGATE.

---

Known limitations

• Holdout accuracy is unmeasured. The win rate above is Adaption's internal metric; the adapter has not been scored against the published 300-row holdout.
• Structural only. The spec says which chart type, axes, grouping and aggregate. It says nothing about whether that is a good way to present the data. Encoding quality is a separate question this does not measure.
• Columns must be in the prompt. The adapter copies from the schema it is given; it has no knowledge of any particular database.
• Academic schemas. nvBench is synthesised from Spider — single-database queries over 152 academic schemas, not production warehouses.

---

Ethics

A visualization spec determines what a reader sees. A wrong aggregate turns a sum into an average; a wrong grouping column merges series that should be separate. Neither error announces itself in the rendered chart.

This adapter is a drafting aid for a pipeline a person reviews, not an autonomous charting agent. The five-field output is deliberately narrow so the spec can be validated against the schema before anything is rendered — column membership is a set lookup, and it should be run.

---

Repositories

Platform	Link
HuggingFace Model	Fernandosr85/nl2vis-chart-spec-adapter
HuggingFace Dataset	Fernandosr85/adaption-nl-to-viz-spec-pairs
Kaggle Model	nl2vis-chart-spec-adapter
Kaggle Diagnostic	nl2vis-zero-shot-headroom

---

Credits

• Fine-tuning platform: Adaption — AutoScientist & Adaptive Data
• Challenge: AutoScientist Challenge 2026 — Data Visualization category
• Source benchmark: nvBench — Luo et al., Synthesizing Natural Language to Visualization Benchmarks from NL2SQL Benchmarks, SIGMOD 2021
• Schemas: Spider — Yu et al., EMNLP 2018
• Related evaluation: VisEval — IEEE TVCG 2024
• Author: Fernando Rodrigues · Kaggle: fernandosr85 · HuggingFace: Fernandosr85

---

Disclaimer

Experimental research artifact submitted to the AutoScientist Challenge 2026 (Data Visualization category). Holdout accuracy is unmeasured. Outputs are structural specifications and require review before rendering or downstream use.

Licensed under the Llama 4 Community License Agreement, inherited from the base model.