Pushback is not an early warning
Monitors that watch for user dissatisfaction assume failing conversations show it before the mistake. On these conversations, they do not.
Pushback before the decisive action
Jev checked every natural user message for a correction, a repeated request, a complaint or a request for a human.
Pushback per message, failed minus passed: −.060 [−.157, +.034]. Failed conversations do not show more pushback before the mistake.
1550 conversations of 4 models. Jev checked every natural user message for a correction, a repeated request, a complaint or a request for a human. The decisive action is the first extra or wrong write action, by the agent or by the customer through a customer tool, or the end of the run if no such write happened.
How good is the pushback check?
Move the threshold to see precision and recall on the validation sample. The finding above uses 0.5, which was fixed before the analysis.
60 user messages, stratified on the highest of all 5 user-message questions (including "says it is done"): 30 scored at least 0.5 there and 30 below. The pushback score uses 4 of them, so fewer than 30 are flagged at 0.5. Labels come from a separate LLM that did not see the scores, not from humans. Precision on a stratified sample is not the population rate, and recall of 1.00 on 10 positives has a wide interval. Filled dots are flagged at the current threshold; click a dot to open its conversation.
Silent failures
Failed conversations with no pushback before the decisive action. Open any one to read it.
Loading runs…