Knowing the task predicts agent failure better than reading the trace
I tested the signals agent-monitoring tools read (tool calls, documents fetched, an LLM judging each step, user complaints) against 7,760 real outcomes from 20 AI models. One simple number about the task beat all of them.
A study by Sudarsh, AI/ML engineer · 20 models · 7,760 customer-support conversations · 97 tasks · Code · Data
Which signal best predicts a failed conversation?
Prediction score: 0.5 is a coin flip, 1.0 is perfect.
Adding the trace to the task's failure rate did not help. It made the prediction slightly worse.
4 models where every signal was run, first 2 user turns.
Some tasks are just hard
Each square is a task; the stronger the blue, the more models fail it. 13 tasks beat every model, every time.
The right documents matter
Same task, same model: in 42 tasks the successful runs had found the right documents more often. In 18, the failed runs had; in 24, no difference.
Users rarely warn you
In 57 of 100 failed conversations, the user never complained before the agent made its mistake.
Before you trust an agent-failure detector, check it against how hard the task is. Most of what it sees may be task difficulty.
Public τ‑bench banking-support data · How it was tested →