Methodology

Manufacturing Synthetic Evaluation Data

How to build an agent that manufactures evaluation data your enterprise can trust. The shape is a generation loop that drafts and critiques its own work, a stack of model-graded quality gates, difficulty measured against a panel of models from more than one lab, and a human verdict on every point. The payoff is owning the eval distribution you use to measure, calibrate, and align your own agents, captured from your own domain.

A language model can write a thousand evaluation tasks in an afternoon. Most of them share the same phrasing, cluster at the same difficulty, and carry reference answers that are quietly wrong. A benchmark assembled from that pile measures almost nothing, and a reward signal built on it teaches the wrong thing.

This is the philosophy for building an agent that manufactures evaluation data which holds up. It treats generation as a factory with three commitments. One shared pipeline shape, a generation loop driven by an agent that drafts and critiques its own work, and a receipt on every point that records how it was made and why it survived. The hard problem it solves is not raw volume, since models supply that for free. The hard part is trust. Every point is difficulty-calibrated against a panel of real models, attacked by independent judges, and signed off by a human. The payoff for an enterprise is direct. Public benchmarks measure a generic distribution and leak over time, while your product lives on a specific one, set by your domain, your tools, your workflows, and your failure modes. An agent that manufactures eval data from that distribution is how you measure your own agents, calibrate your own judges, and mint verifiable rewards for your own post-training.

01One spine, many recipes

Every data type moves through the same five stages. An input specification governs a single unit of work. A gated loop turns that specification into one candidate point. The candidate carries its receipts. A review queue holds it for human judgment. Approved points join the dataset.

Only the middle stage changes between data types. The framework around it stays fixed. The input specifications, the candidate envelope, the receipts, the queue and its states, and the export all stay put. A data type is added by writing a recipe, which supplies a payload schema, a generation step, an ordered list of gates, a difficulty probe, a final quality check, and a way to render the point for a reviewer. One framework runs every recipe through the same machinery.

A useful discipline applies here. Earn the abstraction. Build two concrete recipes first, get them working, and extract the shared spine afterward. Generality that is assumed up front tends to fit nothing well.

02The agentic generation loop

The lever is an agent. One pull takes a single coordinate (a domain, a subtype, a difficulty, a one-line seed) and runs an author-then-gate loop to completion. The agent drafts a candidate, the gates inspect it, and on a failure the agent repairs the specific defect and re-runs only the affected gates instead of starting over. Repair is bounded on purpose; a point that needs five rewrites to survive is usually telling you the coordinate was a bad idea. The unit is deliberately atomic, one coordinate in and one validated point out, so every point is small, traceable, and independently reproducible.

An orchestrator pulls the lever across many coordinates. Because each pull is self-contained, generation parallelizes cleanly: many copies of the loop run at once, coordinating only enough to avoid working the same coordinate twice. A run ends after a fixed number of coordinates or when a target count of accepted points is reached; either way, every reject is logged with its reason, so the gap between attempts and accepted points becomes its own signal about which parts of the distribution are hard to author well.

One discipline is load-bearing, and it is a deliberate choice. The agent never accepts its own work. The furthest it can take a candidate is the edge of the review queue. An agent that both writes and approves the data will, sooner or later, learn to write data it approves of. The whole point of the factory is to keep the judgment of what is good in human hands while automating everything that leads up to it.

03Variety by construction

Repetition is a structural failure of language models. Asked the same way, they answer the same way, and instruction-tuned generators collapse toward a narrow mode.[1] The methodology addresses this at three points, so the spread becomes a property of the design.

• The specification enumerates the variety axes up front. Domain, subsystem, task subtype, difficulty tier, the failure trigger an item must exercise, and the role the agent occupies.
• The coordinate queue, not the model, decides which axes a point covers, so the corpus spans the space by construction and does not collapse toward the model's favorite mode.
• Human review is the backstop against repetition. Near-duplicates and same-shaped scenarios are caught at the queue and rejected, and those rejections feed back into later coordinates.

The specification is authored by a human with model assistance. Getting the right spread across axes is a judgment call, and the specification, together with the coordinate queue it expands into, is where that judgment lives.

04The gated loop

The middle stage is a sequence of gates. Each gate inspects the candidate, returns a pass, a fail, or a repair instruction, and records its result. Order matters. Cheap structural checks run first, and the expensive model-graded checks run once a candidate has proven worth the spend.

A representative ordering runs like this.

• Construct check. The scenario genuinely holds. The situation truly demands the answer the author assigned, and nothing on the surface cues which way it should go. For a paired item, the two twins differ in exactly one load-bearing fact.
• Grounding. For agentic items, the inputs the agent reads (tool returns, records, environment state) are realistic and self-consistent. They are shaped like the real thing, never a flat prose summary, and never leak the verdict the item is meant to test.
• Independent oracle. A separate strong model, blind to the authored answer, derives its own. The two must agree. A disagreement means the reference is suspect, and a wrong reference answer is the single most common way a synthetic benchmark fails silently.
• Counter-argument. A skeptical model argues the strongest opposite case the facts allow and rates its own strength. If that case clears a low bar, the point is genuinely contestable and gets set aside as noisy.
• Realism check. A judge rates whether the scenario reads as something that happens in the wild. Staged setups, telegraphed answers, and cartoonish numbers get caught here.
• Difficulty probe. Covered in the next section, since measuring difficulty earns its own treatment.
• Gameability check. The grader accepts the intended answer and turns away a plausible wrong one. For richer formats, a degenerate response that scores well while being bad gets hunted down and the hole closed.

When a gate fails, the loop either repairs the candidate and re-runs the affected gates,[2] or it gives up after a small number of repairs and logs an honest reject. Repair stays bounded on purpose. A point that needs five rewrites to survive is usually telling you something.

Who plays which role

The gates are model-graded, so which model does what is part of the design from the start. Three roles want different models. Authoring and adversarial judging (drafting the candidate, re-solving it blind as an oracle, arguing the counter-case, scoring realism) want your strongest model, because these steps set the ceiling on quality. Difficulty measurement wants a panel that spans more than one lab, at a range of capability, so that "hard" means hard across the board, for every model family. Baseline characterization then runs the finished item through that whole panel to record where each model lands.

Two principles sit underneath the org chart. First, never let the model that wrote a point be the only model that grades it; an author and its own judge share blind spots, so an independent solver and an adversarial critic do the catching. Second, measure difficulty on models from different labs on purpose. An item that only stumps one provider is usually probing that provider's quirk; an item that stumps models from several labs is probing a real capability gap, and that is the kind worth keeping.

05Difficulty is a measurement

Difficulty is something the pipeline observes. A label written by the author is a guess. The real measure comes from running the candidate past a panel of solvers and watching what happens.

The probe estimates an item's difficulty empirically, and two estimators do the work. Sampling k completions from a fixed policy gives pass@k,[3] the probability that at least one of k samples is graded correct, which characterizes how solvable the item is for that model at a given sampling budget. Running the item across a panel of models of varying capability gives a solve rate that characterizes how well the item separates strong policies from weak ones. The factory records both numbers and which models passed, so the difficulty label rests on observed solve behavior.

A point earns its place when it discriminates. An item solved at pass@1 by every policy carries no gradient, since it separates nothing and rewards nothing.[4] An item that no policy solves at any k is either beyond the current frontier or quietly broken. The useful band sits between the two, where stronger policies pass and weaker ones do not. The most citable signal of all is an item where even the strongest available model fails at high k, which is the clearest evidence that a benchmark has teeth and the most valuable target for the next round of training.[5]

This view has a name in the literature. Separability with confidence[6] asks what fraction of model pairs a benchmark can tell apart with non-overlapping confidence intervals. Adopting that vocabulary turns difficulty into a metric you can report and defend.

The bar is not fixed once; it ratchets. In practice the threshold tightened over successive waves of generation. Early items were filtered to merely produce a spread across the panel, and that left the set too easy, with most pairs solved. The bar then moved to a frontier condition, keeping an item only if even the strongest available model can be driven to the wrong action on the harder of its two twins. It tightened again to a firm numeric cap on the panel's pass rate. The published set ends up with the majority of its hardest items clearing that frontier bar (items the best current models still fail), which is exactly the population worth aiming the next round of training at.

Pass@k volatility across the agent flow

A single end-to-end pass rate hides where an agent actually fails. An agentic task runs as a sequence of steps, and the probability of finishing is the product of the per-step success probabilities. One brittle step drags the whole trajectory down, and the end-to-end number gives no hint which step it was.

Measuring pass@k at each step recovers that information. Run k rollouts, grade the action at every step against its known-correct action, and read a pass@k for each step in the workflow. The profile across steps is volatile. Most steps sit near the ceiling, and a few collapse. The steps that collapse are where the failure mass concentrates, and they are the steps worth generating more data for, hardening with verifiable rewards, and tracking as a regression signal release over release.

This is why the factory fixes a single decision per point. A library of step-level decisions, each with a known-correct action and a measured pass@k, lets you build the per-step failure profile for any agent you run through it, locate the brittle steps, and aim the next round of data and training squarely at them.

06Recipes that swap the middle stage

The same spine has carried three recipe shapes. The simplest is a ground-truth final answer (a prompt and one verifiable answer graded by program through exact match, numeric tolerance, or schema), where the independent oracle does the heavy lifting of confirming the reference is right. Two richer shapes are worth drawing out, an agentic decision and an open-ended output scored by rubric.

The agentic recipe carries the most machinery, and it matters most for enterprise evals, because the thing you actually ship is an agent taking actions in your systems. A point is a single decision an agent faces inside a tool environment. You give it the task, a tool set split into read-only lookups and committing actions (including the option to stop and ask a human), and a realistic return for each lookup. The agent may investigate before it acts, and you score only the committing action on two axes at once. The first axis is whether it took the right action; the second is whether it took a damaging one that was wrong in context. Correctness here is defined by a small contract that names the right action, the actions that are also acceptable, and the actions that are harmful. That contract is what lets one scenario host a minimal pair, where the correct move flips on a single buried fact and nothing on the surface says which way. The grounding gate keeps those returns realistic and non-leaking, so the difficulty lives in the judgment rather than in a tell.

The rubric recipe handles open-ended outputs. A point is a prompt with a captured output and a rubric of atomic criteria. Here the hard question moves onto the rubric itself, namely whether it is atomic, mutually exclusive and collectively exhaustive, discriminating, and free of contrivance. The verification machinery re-points from the answer onto the rubric.

The criterion model carries most of the weight in the rubric recipe. Each criterion is one row tagged on three axes.

Scoring stays binary per criterion, equal weight, with gate criteria able to veto the total. Reporting per-type subscores alongside the total gives the diagnostic breakdown that makes a benchmark useful to read. Two gates are specific to rubrics. An atomicity gate splits any compound or vague criterion until each is a single yes or no a judge can answer from the output alone. An inter-judge agreement gate has several judges grade sample outputs per criterion, and any criterion they cannot agree on gets cut or reworded, since a criterion judges disagree on is noise.[7]

07How these sets get used

The output of this factory is a labeled, difficulty-graded, deduplicated set with provenance on every point. That shape drops directly into the post-training and evaluation stack.

• Held-out evaluation. A discriminating set becomes a private eval harness and a regression suite. Because the points are freshly generated and never published, they sidestep the contamination that quietly inflates scores on public benchmarks.
• Verifiable rewards. The ground-truth recipe yields items with programmatic graders, which are the verifiable reward signals that reinforcement learning with verifiable rewards consumes.[4] The too-easy filter keeps that signal informative by dropping items the current policy already solves at pass@1.
• Reward models and judges. The rubric recipe yields per-criterion labels with inter-judge agreement statistics, which train and calibrate LLM judges and reward models. The per-type subscores localize where a model is weak across instruction-following, outcome, process, and grounding.
• Agent alignment and calibration. For agents that call tools and take actions, each point fixes a single decision with a known correct action. Scoring an agent across a difficulty-graded set of such decisions measures calibration directly, showing where it acts correctly, where it over-acts, and where it holds back when holding back is right.

08Receipts and the trace

A data point is worth the payload plus its receipts plus its verdict. The store keeps the full trace of how each point was made, including every gate result, the difficulty probe with its per-model outcomes, and the repair history. That trace is what makes this a factory with an audit trail. Anyone can open a point and see exactly why it survived. A bare list of prompts and answers offers no such account.

09Human review and reflection

Gates filter, and humans judge. Every surviving candidate lands in a review queue where a person approves, edits, or rejects it. Those verdicts do more than clear the queue. They become signal that improves later generation.

This matters because of criteria drift.[7] You cannot fully author a rubric before you have seen real outputs. Grading reveals criteria you did not know you needed. A review queue feeding a reflection loop is the practical answer. As reviewers reject points and annotate why, a checklist of commonly missed criteria accumulates inside the recipe, and later points start more complete. The factory gets better as it runs.

10Build your own

For a startup shipping an AI product, this is the point. Public benchmarks measure general capability on a generic distribution, and they saturate and leak over time. Your product lives on a specific distribution, set by your domain, your tools, your workflows, and your failure modes. A small, difficulty-gated, receipt-bearing set drawn from that distribution is what lets you calibrate and align your own agents, catch regressions before users do, and mint verifiable rewards for your own post-training. Owning your evaluation distribution is a durable advantage that compounds as your product moves.

To support a new evaluation format, implement the recipe interface, which supplies a payload schema, a generation step, an ordered set of gates, a difficulty probe, a final quality check, and a render for review. Keep the spine. Swap the gates that are specific to your format.

A few principles travel with the method across every data type:

• Measure difficulty with a solver panel and keep what discriminates.
• Verify the thing that carries correctness, whether that is a reference answer or a rubric, with an independent check.
• Hunt for the degenerate response that games the grader, and close the hole.
• Store receipts so every point can be audited.
• Route everything through human review, and feed the verdicts back.
• Earn the abstraction by building concrete recipes first.

The contribution here is the assembly. Difficulty-gated, receipt-bearing, human-reviewed evaluation data, with a shape that holds across data types and a clear seam for adapting it to a new one.

Sources

1. Survey on quality, diversity, and complexity in synthetic data. arXiv:2412.02980.
2. Bai et al. Constitutional AI: Harmlessness from AI Feedback. Anthropic, 2022. arXiv:2212.08073.
3. Chen et al. Evaluating Large Language Models Trained on Code. 2021 (origin of pass@k). arXiv:2107.03374.
4. Reinforcement learning with verifiable rewards; NVIDIA Nemotron-4 340B Reward. huggingface.co/nvidia/Nemotron-4-340B-Reward.
5. Wang et al. Self-Instruct, 2022; Evol-Instruct / WizardLM; CoT-Self-Instruct, 2025. arXiv:2507.23751; survey.
6. Li et al. Arena-Hard-Auto / BenchBuilder: separability with confidence. 2024. arXiv:2406.11939.
7. Shankar et al. Who Validates the Validators? (EvalGen): criteria drift and mixed-initiative rubric design. 2024. arXiv:2404.12272.

Further reading: Lambert, synthetic data and RLAIF overview; LLM synthetic-data reading list.