ServiceNow’s AutoSynthData Lifts Gemma Pass@1 by 7.2 Points

ServiceNow AI says its AutoSynthData pipeline, which turns a model’s failures into new training tasks, raised a Gemma-4-26B-A4B-it model’s mean Pass@1 on the Hybrid domain of EnterpriseOps Gym by 7.2 percentage points, and lifted its mean Pass@1 on the ITSM domain from 18.77% to 27.18%. The figures are ServiceNow’s own, from a Hugging Face post dated 2 October 2026, and cover supervised fine-tuning in two environments.

What ServiceNow Built

ServiceNow says its CoreAI team built AutoSynthData to turn an agent’s capability gaps into training data. It says the pipeline uses a target model’s failures and a stronger teacher’s successes to decide what the model should learn next, then generates and validates new tasks that exercise those capabilities.

ServiceNow defines a task as a system specification, a user prompt and a verifier. It says that as the model improves, the curriculum shifts toward what the model still finds difficult. It illustrates the pipeline with EnterpriseOps Gym, using the released dataset.

How a Failure Becomes a Task

ServiceNow says AutoSynthData first evaluates the target model in the environment with diagnostic tasks and identifies patterns in the tasks it struggles with. A stronger teacher helps characterize which of those tasks are solvable and what successful behavior looks like. The findings are distilled into what it calls sanitized capability specification cards.

It says the generator receives only those cards, not the original evaluation prompts, entities, trajectories or verifier details. Generation runs in two phases. In the target phase, workers generate independent tasks in parallel and each candidate goes through validation, execution, solver evaluation and repair. In the multiply phase, accepted samples are expanded into variants, and ServiceNow says “a multiplied sample cannot seed another multiplied sample.”

The Quality Gates

ServiceNow says it favors tasks the target model solves on no more than one of three trials and the stronger solver solves on at least two of three. It describes a positive gate, which executes the reference trajectory in the environment and checks the resulting state against the verifier, and a negative gate, which, for example, can mutate parts of the expected outcome and confirm those states no longer pass.

Failed candidates go to a critic, which diagnoses the failure and guides repairs, with a fixed limit on retries. A batch-level meta-review then looks for overrepresented task families, missing capability dimensions and low-yield generation patterns, and the controller shifts effort toward the gaps.

The Results

On the Hybrid domain, ServiceNow says it used Gemma-4-26B-A4B-it as the target and Qwen3.8-27B as the teacher. It says AutoSynthData generated 2,000 synthetic training samples in about 18 hours, and that the best checkpoint was epoch 5.

ServiceNow says the synthetic SFT checkpoint improves mean Pass@1 by 7.2 percentage points, a 35% relative improvement, and raises verifier success from 63.01% to 68.55%. It says this closes 59% of the original Pass@1 gap between Gemma and the reference model.

On the ITSM domain, ServiceNow says it used the same Gemma model as the target and DeepSeek-V4.1-Flash as the teacher. It says the pipeline generated 1,994 samples in 66 hours, longer than the Hybrid run, “primarily because the ITSM run used a larger teacher model and preceded pipeline optimizations that improved throughput.” Synthetic SFT raised mean Pass@1 from 18.77% to 27.18%.

The Structural Read

The post describes three moving parts: a diagnosis step that finds where the target model fails and how a stronger teacher succeeds, a generation step that works from capability cards instead of the original evaluation tasks, and a checking step made of positive and negative gates, a critic and a batch-level review. The results section reports end-to-end gains.

The two results are not like for like. Hybrid is reported as verifier success, 63.01% to 68.55%, alongside a 7.2-point gain in mean Pass@1. ITSM is reported as mean Pass@1, 18.77% to 27.18%. The teachers differ, Qwen3.8-27B for Hybrid and DeepSeek-V4.1-Flash for ITSM, and ServiceNow says the ITSM run was slower because it used a larger teacher and preceded pipeline optimizations.

ServiceNow says it built the capability cards from runs of the target model and the teacher on the evaluation tasks, and that the generator received the cards but not the original evaluation prompts, entities, trajectories or verifier details.

Three Implications

A TEACHER IS PART OF THE RECIPE ServiceNow says it favors tasks the stronger solver solves on at least two of three trials, and that the stronger teacher demonstrates a successful trajectory for each task for supervised fine-tuning. The recipe therefore starts from a stronger model than the one being trained, and ServiceNow names the teachers it used.

THE VERIFIER CARRIES THE WEIGHT ServiceNow lists three properties a verifier should have, consistency, soundness and completeness, and says a lax verifier can reward incorrect behavior while an overly restrictive one can penalize valid solutions. Much of the pipeline it describes is checking: two verification gates, a critic and a batch review.

WHAT REMAINS UNKNOWN The post reports gains from supervised fine-tuning in two environments and names reinforcement learning as something it plans to test. It states the Hybrid gain as 7.2 points and 35% without the mean Pass@1 levels. This publication did not read the model or dataset cards or the benchmark paper.

What Is Not Established

ServiceNow says its experiments focus on supervised fine-tuning and that it plans to test the approach beyond SFT, including reinforcement learning. It says the Hybrid result “demonstrates improvement in EnterpriseOps Gym Hybrid, the environment used for this experiment.”

The text of the post states the Hybrid gain as 7.2 points and 35% but does not give the mean Pass@1 before and after. This publication read the post text only, did not read the model or dataset cards or the benchmark paper, and did not seek a response from ServiceNow.

The Bottom Line

ServiceNow says AutoSynthData generated 2,000 synthetic training samples on Hybrid and 1,994 on ITSM from a model’s own failures and a stronger teacher’s successes, and that supervised fine-tuning on them raised Gemma’s mean Pass@1 by 7.2 percentage points on Hybrid and from 18.77% to 27.18% on ITSM. The results are ServiceNow’s own, in two environments, and it has not yet tested reinforcement learning.