Method and limits

τ-bench (tau2-bench) public leaderboard trajectories for the banking_knowledge domain: 97 tasks, about 700 policy documents, an LLM-simulated customer (gpt-5.2), and grading by the final database state. 20 models × 97 tasks × 4 trials = 7,760 conversations.

Labels re-scored. Several older trajectory files carried rewards from before tau2-bench v1.0.1 fixed banking grading. Every file was re-scored with the v1.0.1 grading code; re-scored pass rates match the leaderboard within 0.3 points. 6 files were excluded: 4 that v1.0.1 cannot load and 2 that stay off the leaderboard after re-scoring. The pooled results repeat on the 12 files whose original labels already matched.

Models: Claude Opus 4.5, Claude Opus 4.6, Claude Opus 4.7, Claude Opus 4.8, Claude Opus 5, Claude Sonnet 4.5, Gemini 2.5 Pro, Gemini 3.1 Pro Preview, GLM-5.2, GPT-5.2, GPT-5.2, GPT-5.4, GPT-5.5, Grok 4.1 fast, Grok 4.2, Grok 4.5, Grok 4 fast, Inkling, Kimi K3, Qwen 3.8 Max.

Trace facts (code, no model): tool counts, search and write calls, required documents surfaced, in context and opened, repeated identical calls, transfers to a human, message lengths. Computed on the first k user turns (k = 2, 4, 6, all), using only messages up to turn k.

Step judgments (TypeSafe Jev 1.14, 4 models): 8 questions on each of the first 6 agent steps. The task given to Jev is the customer's first message; no ground truth is sent.

Pushback checks (Jev 1.14, 4 models): one request per natural user message, with the previous agent and user messages. Pushback is the highest of corrects, repeats, complains and asks for a human.

Decisive point: the first extra or wrong write action (by the agent, or by the customer through a customer tool), or the end of the run if no such write happened.

Jev usage: 17,160 calls, $1.50 in total.

Logistic regression with fixed C = 1 and no tuning. Cross-validation is grouped by task, so no task is in both the training and the test fold. The task prior for a run is that task's pass rate among the other models. Adding weak features to an untuned model can lower AUROC slightly; that is why some changes are negative.

95% intervals come from 2,000 bootstrap resamples of tasks (seed 0).

• Simulated users. The customer is an LLM that follows the task script; real customers differ, and the simulator cannot know bank policy, so policy mistakes can go unnoticed by design.
• The task prior needs repeated tasks. It exists in eval and regression suites, not for a new task in production.
• One domain (banking support) and one benchmark.
• The retrieval result is an association, not a cause.
• Validation labels come from an LLM, on 60 stratified messages; the 0.8 threshold was chosen on that same sample.
• The "opened" document measure is loose: any later mention of a document id counts.
• Prediction uses logistic regression only; a stronger model might extract more from the trace.

• Failure attribution in the sense of Who&When (ICML 2025) finds which agent and which step caused a failure. This study asks a different question: whether trace signals predict failure at all, against a task-difficulty baseline.
• EarlyEval evaluates failure predictors on new agents but seen tasks, without a per-task baseline. On τ-bench banking that baseline alone reaches AUROC 0.86.
• TRACE detects user dissatisfaction and traces it back to a cause, evaluated on 75 synthetic traces. Here, pushback flags successful runs at least as often as failing ones before the mistake.
• τ-Knowledge (the benchmark) and Lost in Simulation (limits of simulated users).

Every number on this site is exported from the analysis tables by python -m analysis.export_site. The full pipeline, from downloading trajectories to re-scoring and the Jev calls (cached by payload hash), is in the repository README. The conversations with Jev annotations are a Hugging Face dataset.