Building benchmarks from real work: saturation, mocks and contamination
1. Why public benchmarks are saturating
A benchmark is valuable only insofar as it separates models of differing capability. When leading models score close to one another and approach the ceiling on a benchmark, it stops telling us much that is new. This is what people usually mean by saturation.
We see at least three sources of saturation. First, the task distribution is detached from real work: many benchmarks are written by researchers or collected from public competitions, with clear task boundaries and tidy inputs and outputs, whereas real work is full of incomplete requirements, legacy code and cross-system dependencies. Second, whatever is published gets absorbed: once tasks are public, they may enter model corpora directly or indirectly, so a higher score does not necessarily mean higher capability. Third, outcome scoring is too coarse: looking only at whether the final result passes makes it hard to tell a run that was on track but unfinished from one that went wrong from the start.
Rather than build yet another, harder question bank, we want to know whether evaluation tasks can come directly from work that actually exists, and whether the way they are released can control how quickly they are absorbed.
2. Turning legacy enterprise systems into RL environments
Legacy systems inside enterprises, such as approval flows, reporting tools and ageing business back-ends, are exactly what agents most often face in real deployments. But they cannot be used for evaluation as they are: they are connected to production systems, depend on internal network services, and their state cannot be reset. Our approach is to mock them:
- Boundary identification: map the system’s external dependencies (databases, external APIs, authentication, file storage) and decide which must be replaced and which can be kept.
- Content replacement: replace real content with structurally consistent, semantically plausible synthetic content, ensuring it contains no personal information or trade secrets while preserving the business constraints that make the task hard.
- Dependency stubbing: write behaviour-consistent stub services for external interfaces so the system can run fully inside an isolated container.
- Resettable state: start every task episode from the same snapshot so results are comparable across models and runs.
- Executable verification: define programmatically checkable completion conditions for each task, supplemented where needed by expert-written rubrics for process scoring.
After these steps, a legacy system becomes an interactive environment with a uniform interface (reset, act, observe) that can serve both as an evaluation environment and as a reinforcement-learning environment.
3. Managing contamination
Contamination cannot be eliminated entirely, but it can be managed. We plan to:
- Separate a public subset from a private test set: publish only a small number of sample tasks to illustrate format and difficulty, and run the main evaluation on the private set.
- Version and rotate: release benchmarks in versions, replace tasks regularly, and state the version when reporting results.
- Keep sources traceable: record each task’s source type and build date so it is possible to analyse afterwards whether a result may have been affected by contamination.
- Score the process, not just the outcome: combine step-level process scoring to reduce inflated scores from memorised answers.
4. Open questions
How much of a real system’s difficulty does mocking preserve? How should process scoring balance expert judgement against reproducibility? How can a private test set stay credible while remaining open to external scrutiny? These are the questions we aim to answer next, and we welcome discussion with others working on them.
— Seven Eighths Lab