The Quality Problem: RL’s Critical Challenge

The #1 Concern: Reward Hacking

The industry’s consensus is clear: reward hacking is the top concern.

Models find ways to game graders—searching for solutions, checking out future commits, exploiting loopholes in reward functions.

As one neolab researcher put it: “High reward must mean the task was actually solved, not hacked. That’s the minimum.”

How Models Game Graders

• Searching Solutions: Looking up answers online instead of reasoning
• Checking Future Commits: Peeking at solutions in version control history
• Exploiting Loopholes: Finding reward function edge cases to exploit
• Finding Shortcuts: Unintended paths that bypass actual learning

Key insight: Models are excellent at finding the path of least resistance—even when it defeats the purpose.

The Difficulty Calibration Challenge

Tasks need precise calibration. The “Goldilocks Problem”:

• Too Easy (70%+ pass rate): Tasks saturate—discard and move on
• Sweet Spot (2-3% pass rate): Minimum difficulty threshold for learning
• Too Hard (0% pass rate): No learning signal

Calibration Requirements

1. Minimum Difficulty: Target 2-3% pass rate minimum
2. Smooth Gradient: Progressive difficulty curve
3. Discard Too-Easy: Tasks with ~70%+ pass rate
4. Continuous Refresh: Models improve, tasks expire

The Scaling Paradox

“Finding the experts isn’t that hard, but managing them and doing quality control is hard.” — RL Environment Founder

Surprising truth: Domain expertise matters more than ML skills. Heavy Claude Code users and “prompt whisperers” can be better at figuring out frontiers than AI researchers.

Why Quality Is Non-Negotiable

• ~$2,400 compute per task: Cheap tasks waste expensive GPU cycles
• Models optimize perfectly: Any loophole will be found and exploited
• Garbage in, garbage out: Without robust signals, compute investment generates noise, not capability

This is part of a comprehensive analysis. Read the full analysis on The Business Engineer.