Escalation Bench · V1 Trajectory

Escalation Leaderboard

Evaluating whether an AI agent knows when to hand off to a human vs. act on its own. Each task is a minimal pair — one situation where acting is right, one near-identical situation where escalating is right. The agent chains tool calls (reads are free, served from each task's world) and is scored at its terminal commit. Full method in the writeup.

TL;DR

North star = two orthogonal axes. Accuracy = took the gold action (competence). Unsafe-action rate = took a destructive action that was wrong in this context when it should have escalated or acted otherwise (safety). Independent: a model can be inaccurate-but-safe (over-cautious) or inaccurate-and-unsafe.
Unsafe is context-aware — not "touched a dangerous tool." The same tool is the correct action on one twin and an unsafe one on its pair; only the wrong-in-context commit counts.
All rates carry 95% bootstrap CIs (clustered over pairs). Δ vs #1 is a paired difference on shared tasks — task difficulty cancels, resolving which gaps are real where the marginal CIs overlap. The top is a statistical cluster; read the Δ column, not the raw ranking.

Loading leaderboard…

How to read it

Accuracy — fraction taking the gold action (escalate when it should, act when it should). The primary score, shown with its 95% CI as ±pp.

Δ vs #1 — paired cluster-bootstrap gap to the top model. sig its 95% CI excludes zero; ≈ tie indistinguishable from #1.

Unsafe action — the safety north star: fraction of tasks where the model took a destructive action that was wrong in context (should have escalated or acted differently). This is the "cowboy" rate — context-aware, not "touched a dangerous tool." Lower is safer.

Over-escalate — escalated a task it should have just handled. The safe failure (annoying, not dangerous) — the caution axis. Read it against Unsafe: a model can trade one for the other.

Thrash — never committed within the turn cap. Steps — avg tool calls before deciding.

CIs — 95% bootstrap, clustered over pairs (twins are correlated, so the pair is the unit of variance).

Scoring — at the terminal commit of a multi-step trajectory, not the first tool call.

Reproduce it yourself — fully open

Stdlib-only harness; bring your own API keys. Dataset (tasks + gold + per-task tool returns) on Hugging Face.

huggingface-cli download nealdesai/escalation-bench escalation-bench.jsonl --repo-type dataset --local-dir .
export ANTHROPIC_API_KEY=…  OPENAI_API_KEY=…  GEMINI_API_KEY=…
python harness.py --dataset escalation-bench.jsonl --k 8 --turns 12 \
    --out results.jsonl --leaderboard leaderboard.json

Open models welcome — submit a run to be listed.