Wiki
Core11 min read

Evaluating trajectories

Agents are non-deterministic, so you test paths, not answers. Run the task many times, classify how it fails, and report rates — pass@k and pass^k tell different stories.

A chat model's output is one string you can compare to an answer. An agent's output is a trajectory — a path of thoughts, tool calls, and observations — and the same input can take different paths twice. That makes evaluation different in kind: you are not checking an answer, you are sampling a process and measuring how often, and how, it works.

Start here

One successful run is an anecdote. A working agent is one that succeeds reliably — so you run the same task many times, look at the distribution of outcomes, and classify the failures. "It worked when I tried it" is not an evaluation.

One task, 3 runs — same input, different paths

run 1: read → sort → calc → answerpass · clean
run 2: read → calc (wrong) → retry → answerpass · recovered from an error
run 3: read → answerfail · premature stop

pass rate

2/3

pass@3 (any)

yes

A single green run proves little when the next run might loop or stop early. Test the trajectory: run it many times, classify *how* it fails (premature stop, runaway loop, wrong tool), and report a rate. pass@k ("at least one of k works") and pass^k ("all k work") tell very different stories about reliability. Runs are illustrative.

Slide the number of runs and watch a task that passed become a task that passes 60% of the time.

Test paths, not answers

  • Sample, don't spot-check — run each case several times; a non-deterministic system needs a rate, not a result.
  • Classify failures — premature stop, runaway loop, wrong tool, budget exhaustion. The kind of failure tells you what to fix; the rate tells you whether it matters.
  • Check the process — a right answer via a wrong path (a lucky guess, a skipped verification) will not generalise, so score the trajectory too, not just the endpoint.

pass@k and pass^k

  • pass@k — does at least one of k attempts succeed? The optimistic view: can it do the task at all?
  • pass^k — do all k attempts succeed? The reliability view: can you depend on it?

An agent can have high pass@k and low pass^k — capable but flaky — and for anything automated, reliability is the number that matters.

Careful

Evaluate on cost, not just success. An agent that solves the task 5% more often for 10× the tokens is usually a regression, and one that passes your eval but cannot be traced, capped, or rolled back is a liability. Track success, cost, and latency together — a benchmark win that ignores the other two is how "improvements" make production worse.

Check yourself

Eduspheria wiki · Agentic AI, Evaluation & observability

0 / 5 answered

  1. 1What does pass^k measure?
    Multiple choice
  2. 2An agent can have high pass@k and low pass^k — capable but flaky.
    True / false
  3. 3What is the name for the full path of thoughts, tool calls, and observations an agent produces?
    Short answer
  4. 4An agent succeeds on 3 of 5 independent runs. What is its observed success rate as a decimal?
    Numeric answer
  5. 5Why score the trajectory and not just the final answer?
    Multiple choice

Next: the public benchmarks, and what they do and don't tell you.