Synthetic Eval Datasets: Skip the Framework, Just Prompt

Why synthetic eval datasets?

If you're building an LLM-powered pipeline — RAG, agents, tool-use chains — you need evaluation data to catch regressions and measure quality. Real user queries are ideal but slow to collect, expensive to label, and often unavailable early in development. Synthetic datasets let you generate hundreds of targeted test cases on demand, covering edge cases and failure modes you'd wait months to see organically.

The open question isn't whether to use synthetic data, but how to generate it. Most teams either reach for a framework (DeepEval, promptfoo) or write a quick prompt. We wanted to know if that choice matters.

Method

Human annotation processes typically end with a quality review step — a separate reviewer inspects the data, compares samples, and flags what doesn't hold up. We mimic this process by using LLMs as judges: given two datasets, the judge reads both and picks the better one on each quality criterion. This "pairwise comparison" approach measures the dataset directly — unlike running queries through answer models and scoring responses, which mixes up dataset quality with how good the answer model is.

We crossed 3 generation methods with 3 generator models to produce 9 datasets of 20 queries each. Every pair of datasets (36 pairs) was evaluated by all 3 models as judges — 108 comparisons total. To avoid judges favoring whichever dataset appears first, we swapped the presentation order.

Generation methods

Generator models

All three models also served as judges, creating a 3x3 judge matrix that lets us measure self-preference bias.

Judging criteria

Each judge receives the full 20 queries from both datasets and scores them on a 5-point comparative scale — from A>> (strongly prefer A) through tie to B>> (strongly prefer B). The judge scores five dimensions:

1. Diversity — topic spread, lexical variety, structural variety
2. Difficulty — challenges the model, exposes failure modes and edge cases
3. Validity — well-formed, unambiguous, actually testable by an LLM
4. Realism — reflects real-world scenarios practitioners would encounter
5. Overall — a holistic judgment of which dataset is the better eval benchmark

The overall score is a direct LLM judgment, not a weighted average of the four criteria. The judge is free to weigh trade-offs as it sees fit — which is exactly what makes the self-preference analysis interesting later. All judges run at temperature=0.0 and return structured JSON with a reasoning field explaining their verdict. The full judge prompt is included at the bottom of this post.

Results: generator model matters more than method

ELO rankings

ELO ratings — borrowed from chess rankings — give each dataset a single score based on its head-to-head wins and losses. Higher is better; 1000 is the starting point.

Two clear tiers: GPT-5.4/Sonnet 4.6-generated datasets cluster at 1065–1143 ELO, while DeepSeek V3.2-generated datasets fall to 784–920.

The model you use to generate matters more than how you prompt it. The gap between the best and worst generator (233 ELO) is nearly double the gap between the best and worst method (121 ELO). If you're investing in eval dataset quality, upgrade your generator model first.

Per-criterion breakdown

Each dataset appears in 8 matchups (against every other dataset), each judged by all 3 models — 24 judgments total. For each criterion, the judge's verdict maps to a score: strongly prefer (+2), slightly prefer (+1), tie (0), slightly prefer opponent (−1), strongly prefer opponent (−2). The per-criterion score is the mean across all 24 judgments, so +1.5 means judges consistently preferred that dataset on that dimension.

Breaking it down by criterion reveals where each approach actually wins and loses:

• Custom prompting dominates realism — the dimension most tied to practical usefulness
• DeepEval's one clear advantage is difficulty — it generates harder, more failure-provoking questions
• Diversity is a wash — no method or model separates meaningfully
• DeepSeek V3.2 is the validity bottleneck regardless of which method wraps it

But can we trust the judges?

Every judge in this experiment also generated one of the datasets it's evaluating. If a model is biased toward its own output, its datasets get inflated scores.

Across all 108 comparisons, judges agree on the winner 73% of the time — but agreement doesn't rule out shared bias. The judge × generator matrix reveals the tilt:

GPT-5.4-as-judge shows the strongest self-preference: it rates its own model's datasets +0.88 on average but rates other models' datasets -0.44 — a gap of 1.31 points on the 5-point scale. Sonnet 4.6 shows a similar but smaller gap of 0.87. DeepSeek V3.2 doesn't show positive self-preference, but that's because its datasets are genuinely weaker — all judges agree on that.

What happens when we remove self-judgments?

If the bias is real, removing self-judgments should change the rankings. It does — but not as much as you'd expect.

GPT-5.4 and Sonnet 4.6 datasets drop 55–64 ELO — their scores were partly inflated by self-preference. DeepSeek V3.2 datasets rise 26–83 ELO — penalized by lacking a friendly self-judge. The ranking order mostly holds, but the spread shrinks from 359 to 213 ELO. Roughly 40% of the apparent quality gap between models was judge bias, not actual quality difference.

Practical takeaways for building eval datasets

1. Generator model > generation method. If you're building eval datasets, invest in the best generator model you can afford. The prompting strategy matters less — though method choice still shapes which quality dimension you optimize for (DeepEval produces harder questions, custom prompting produces more realistic ones).

2. LLM judges have measurable self-preference. GPT-5.4-as-judge inflates GPT-5.4-generated content by ~1.3 points on a 5-point scale. Always use cross-model judges or remove self-judgments from analysis.

3. A coding agent's one-shot prompt matched or outperformed both frameworks. No multi-step synthesis, no evolution loops — just a prompt generated by Claude Code. It scored highest on realism, the criterion most tied to practical utility.

Appendix: all 9 datasets with per-judge scores

Each card below contains the full set of 20 queries and a breakdown of how each judge scored the dataset across all pairwise matchups.

The "custom" prompt that beat the frameworks

Since the whole point of this post is that a plain prompt works as well as a framework, here's the exact prompt we used — generated by Claude Code in one shot. No multi-step synthesis, no evolution loops, no schema validation.

That's it. This prompt, fed to GPT-5.4, produced the highest-ranked dataset in the experiment. The same prompt fed to Sonnet 4.6 produced the second-highest. The model did the work — the prompt just pointed it in the right direction.

The DeepEval Synthesizer setup

DeepEval's Synthesizer generates test cases from a StylingConfig that describes the scenario, task, and expected I/O format. Under the hood it runs multi-step synthesis with an evolution loop to diversify the outputs.

The promptfoo-style prompt

The promptfoo method uses a simpler structured prompt that asks for a JSON array directly — similar to how promptfoo's built-in dataset generation works.

The judge prompt

Each pairwise comparison uses the following prompt. The judge sees all 20 queries from both datasets, scores 4 criteria plus an overall judgment, and returns structured JSON with reasoning. Position bias is mitigated by deterministically swapping which dataset appears as "A" vs "B" based on a SHA-256 hash of the pair names.

Note that the overall field is a direct holistic judgment by the LLM — it's not computed from the four criteria. This matters: when we analyze self-preference bias, the overall score reveals how judges weigh trade-offs differently, not just how they rate individual dimensions.