Wiki
Core10 min read

Agent benchmarks

SWE-bench, WebArena, GAIA, AgentBench, τ-bench — what each actually measures, what it misses, and why a public benchmark becomes a target.

Public benchmarks are how the field compares agents, and they are useful — but each one measures a specific, narrow slice of capability, and the number on the leaderboard is not the number you care about. Read the benchmark's scope and caveat before you read its score.

Start here

A benchmark is a proxy for the thing you actually want. It is chosen to be measurable and comparable, which makes it different from your task by construction. Use benchmarks to compare approaches; use your own cases to decide what to ship.

What agent benchmarks actually measure

measures

fixing real GitHub issues, judged by the project's tests.

caveat

Solutions can leak into training data, and passing tests is not the same as a maintainable fix.

A benchmark is a proxy, and a public one becomes a target — Goodhart's law applies to models as much as to metrics. Use them to compare approaches, then build your own eval from your own tasks, because the only benchmark that predicts your production behaviour is the one made of your production cases.

Select each benchmark to see what it measures and where it misleads.

The landscape

  • SWE-bench — resolve real GitHub issues, judged by the repo's own tests. Strong signal for coding agents; vulnerable to test leakage and to "passes the test, not maintainable".
  • WebArena — multi-step tasks in sandboxed, self-hosted sites. Tests real browser behaviour in a controlled web.
  • GAIA — general assistant questions requiring tools and multi-step search. Tests composition; small and public enough to be memorised.
  • AgentBench — OS, database, web, and game environments in one suite. Broad, but environment-specific skills may not transfer.
  • τ-bench — tool + agent + simulated-user dialogue in a domain, with pass^k to expose flakiness.

Why a public benchmark becomes a target

Once a benchmark is the scoreboard, effort flows toward the score. Test cases leak into training data; solutions overfit to the test split; metrics get gamed. This is Goodhart's law, and it is not a reason to ignore benchmarks — it is a reason to treat them as one input among several.

Careful

Build your own eval. The only benchmark that predicts your production behaviour is one made of your production cases, with your tools, your failure modes, and your cost and latency limits. Keep it small, keep it current, and re-run it on every change — including prompt and model version bumps, which silently change behaviour.

Check yourself

Eduspheria wiki · Agentic AI, Evaluation & observability

0 / 4 answered

  1. 1Which benchmark resolves real GitHub issues and is judged by the repository's own tests?
    Multiple choice
  2. 2A public benchmark score is the number that matters most for your production task.
    True / false
  3. 3What law says that once a measure becomes a target, it ceases to be a good measure?
    Short answer
  4. 4What is the only benchmark that predicts your production behaviour?
    Multiple choice

Next: grading the things that have no single right answer.