Retrieval within a task

Within the same task and model, runs that got all required policy documents into the agent's context failed less often.

Passed vs. failed runs of the same task

One bar per task with mixed outcomes. Height: share of passed runs with all required documents in context, minus the same share for failed runs (averaged over models). Hover a bar for the task.

task_061: +1.000 (1 model with mixed outcomes)task_050: +0.809 (7 models with mixed outcomes)task_032: +0.762 (7 models with mixed outcomes)task_044: +0.667 (4 models with mixed outcomes)task_099: +0.646 (8 models with mixed outcomes)task_081: +0.583 (2 models with mixed outcomes)task_031: +0.556 (6 models with mixed outcomes)task_051: +0.556 (6 models with mixed outcomes)task_012: +0.514 (12 models with mixed outcomes)task_004: +0.500 (6 models with mixed outcomes)task_088: +0.500 (2 models with mixed outcomes)task_033: +0.476 (14 models with mixed outcomes)task_037: +0.449 (13 models with mixed outcomes)task_035: +0.444 (3 models with mixed outcomes)task_005: +0.426 (9 models with mixed outcomes)task_063: +0.375 (8 models with mixed outcomes)task_073: +0.375 (8 models with mixed outcomes)task_003: +0.333 (14 models with mixed outcomes)task_026: +0.333 (3 models with mixed outcomes)task_052: +0.333 (5 models with mixed outcomes)task_045: +0.309 (7 models with mixed outcomes)task_055: +0.288 (11 models with mixed outcomes)task_057: +0.273 (11 models with mixed outcomes)task_100: +0.271 (8 models with mixed outcomes)task_054: +0.229 (8 models with mixed outcomes)task_010: +0.151 (11 models with mixed outcomes)task_018: +0.150 (10 models with mixed outcomes)task_022: +0.150 (10 models with mixed outcomes)task_021: +0.148 (9 models with mixed outcomes)task_019: +0.143 (7 models with mixed outcomes)task_025: +0.139 (12 models with mixed outcomes)task_016: +0.111 (18 models with mixed outcomes)task_043: +0.111 (6 models with mixed outcomes)task_038: +0.095 (7 models with mixed outcomes)task_041: +0.067 (5 models with mixed outcomes)task_071: +0.063 (8 models with mixed outcomes)task_014: +0.059 (14 models with mixed outcomes)task_020: +0.056 (3 models with mixed outcomes)task_058: +0.037 (9 models with mixed outcomes)task_028: +0.033 (10 models with mixed outcomes)task_008: +0.030 (11 models with mixed outcomes)task_017: +0.028 (6 models with mixed outcomes)task_001: 0.000 (7 models with mixed outcomes)task_002: 0.000 (5 models with mixed outcomes)task_006: 0.000 (4 models with mixed outcomes)task_036: 0.000 (10 models with mixed outcomes)task_039: 0.000 (1 model with mixed outcomes)task_040: 0.000 (4 models with mixed outcomes)task_046: 0.000 (1 model with mixed outcomes)task_064: 0.000 (9 models with mixed outcomes)task_075: 0.000 (11 models with mixed outcomes)task_076: 0.000 (8 models with mixed outcomes)task_077: 0.000 (3 models with mixed outcomes)task_078: 0.000 (1 model with mixed outcomes)task_080: 0.000 (3 models with mixed outcomes)task_085: 0.000 (5 models with mixed outcomes)task_087: 0.000 (2 models with mixed outcomes)task_089: 0.000 (8 models with mixed outcomes)task_090: 0.000 (11 models with mixed outcomes)task_094: 0.000 (8 models with mixed outcomes)task_095: 0.000 (5 models with mixed outcomes)task_096: 0.000 (6 models with mixed outcomes)task_097: 0.000 (6 models with mixed outcomes)task_098: 0.000 (7 models with mixed outcomes)task_101: 0.000 (5 models with mixed outcomes)task_102: 0.000 (9 models with mixed outcomes)task_034: -0.013 (13 models with mixed outcomes)task_024: -0.026 (13 models with mixed outcomes)task_093: -0.037 (9 models with mixed outcomes)task_049: -0.042 (4 models with mixed outcomes)task_070: -0.050 (10 models with mixed outcomes)task_056: -0.067 (10 models with mixed outcomes)task_062: -0.067 (5 models with mixed outcomes)task_015: -0.083 (4 models with mixed outcomes)task_023: -0.083 (8 models with mixed outcomes)task_079: -0.111 (3 models with mixed outcomes)task_048: -0.133 (5 models with mixed outcomes)task_007: -0.135 (16 models with mixed outcomes)task_072: -0.185 (9 models with mixed outcomes)task_047: -0.194 (6 models with mixed outcomes)task_059: -0.278 (6 models with mixed outcomes)task_029: -0.333 (1 model with mixed outcomes)task_060: -0.333 (1 model with mixed outcomes)task_074: -0.333 (5 models with mixed outcomes)↑ Passed runs had the documents more often↓ Failed runs had them more often
Tasks, passed higher / failed higher / tied
42 / 18 / 24 · sign test p = 0.0027
Change in failure risk, same model and task
−.351 [−.464, −.232]
Mantel–Haenszel, strata with mixed outcomes
Over all strata
−.156 [−.217, −.103]
Includes strata where all runs pass or all fail

How to read this

A raw comparison across tasks misleads: tasks that need many documents are both harder and less often fully retrieved. Comparing runs of the same task and model removes that. Tied tasks are those where passed and failed runs had the documents equally often.

The risk change is in failure probability, all documents in context minus not. A negative number means fewer failures when the documents were there. "In context" means the document body entered the agent's context; "opened" is stricter and means the agent deliberately read it. The stricter measure points the same way but is weaker.

This is an association. Runs that retrieve well may also be runs where the agent understood the request; only supplying the correct documents in an experiment would show cause.

Strata are model and task pairs. Brackets: 95% intervals, bootstrap over tasks.