The task comes first
Which task an agent faced predicts its failure far better than which model it was, or anything in its trace.
Pooled over all 7,760 runs. Task difficulty is the task's failure rate among the other 19 models; model identity is the model's own failure rate. Brackets: 95% intervals, bootstrap over tasks.
Every run, by model and task
Vertical bands are tasks that almost every model fails or passes. Click a cell to see its 4 runs, then open a conversation.
Hover a cell to read it. Click a cell to see its runs.
Adding trace signals to task difficulty, by model
The same comparison as on the Overview, per model. Select a model to see that the pattern holds for each one.
- Adding trace facts to task difficulty: −.022 [−.036, −.010]
- Adding trace facts to task difficulty and own trials: −.009 [−.015, −.003]
AUROC for predicting failure, logistic regression, cross-validated by task. Lines are 95% intervals from a bootstrap over tasks. The dashed line marks task difficulty.
"This model's other trials" is a second task-level prior: the same model's results on its other 3 attempts at the task (leave-one-trial-out). It is not a trace signal, which is why it can beat the first prior.