Skip to content
Here Be Dragons

Benchmark Contamination

Benchmark scores are not reliability guarantees

Published benchmark numbers are useful, but they are not production reliability guarantees. Coding benchmarks often live in public repositories, papers, tutorials, blog posts, solution writeups, and model-evaluation harnesses. That makes them easy to accidentally include in training data.

If a model has seen a benchmark problem, its score may measure memorization, near-duplicate exposure, or benchmark-specific adaptation instead of general software skill. The result is familiar: the leaderboard looks strong, then the model struggles with a messy internal codebase, stale docs, ambiguous requirements, flaky tests, and long-running context.

Benchmark contamination is not the only reason coding agents fail. It is one reason public scores can feel disconnected from daily engineering reliability.

Key ideas

  • Public benchmarks leak: HumanEval, MBPP, SWE-bench tasks, issue histories, and solution discussions are widely mirrored and discussed.
  • Moving targets help: Benchmarks with fresh problems reduce exposure risk, but they still may not match your stack or workflow.
  • Long-horizon work is different: A single function problem is not the same as a multi-file change with tests, migrations, review comments, and product ambiguity.
  • Contamination can hide stale API knowledge: A model may do well on known benchmark tasks while still writing outdated framework code.
  • Private evals matter: Your own hidden tasks are harder to game and closer to what you need the tool to do.

What contamination looks like

Contamination can happen at several levels:

  • exact benchmark problem in training data
  • benchmark solution in training data
  • near-duplicate problem from a tutorial or fork
  • issue and pull request history from the same repository
  • evaluation prompts leaked into examples
  • model fine-tuning based on public benchmark feedback

The model may not “remember” the problem in a human sense. It only needs enough exposure to learn the answer pattern.

Building better local evals

Use public benchmarks for broad comparison, then build a private eval set for deployment decisions:

  • take real bugs, small feature requests, refactors, and doc tasks from your own repos
  • freeze the starting repository state
  • define the expected behavior and validation commands
  • include hidden tests when possible
  • score objective outcomes before subjective prose quality
  • rerun the same evals on model, prompt, tool, and harness changes

Private evals should cover the work you actually delegate. If the agent writes migrations, include migrations. If it updates docs, include broken links and formatting rules. If it reviews security-sensitive code, include vulnerability cases.

What to measure

Go beyond “did it answer correctly”:

  • tests passed without false claims
  • files changed were in scope
  • dependencies were not invented
  • security checks stayed clean
  • links and docs remained valid
  • the agent recovered from tool errors
  • the agent asked for clarification when requirements were under-specified

Those signals predict production usefulness better than a generic coding score alone.

Common failure modes

  • Leaderboard chasing: A team picks a model because of a public score, then ignores local failure modes.
  • No fresh tasks: The eval set becomes known to the team and eventually leaks into examples or prompts.
  • Only happy paths: The eval set skips flaky tests, ambiguous requirements, and missing files.
  • Scoring by vibes: Reviewers rate fluent answers highly even when the code does not run.
  • No regression tracking: A new model improves one benchmark but breaks the team’s actual workflow.

References

Why the name "Here Be Dragons"?