Microsoft and Hugging Face introduced ThinkingBox, an evaluation that runs each of 507 stateful tasks 20 times and checks final database values and side effects. In a common comparison of 12 models, 67.24% of attempts that failed verification had terminated normally and called state-changing tools, yet produced incorrect results. Opus 5.5 achieved 67.16% single-attempt success, but succeeded on all 20 runs for only 47.53% of tasks. Kimi-K3 succeeded at least once on 93.89% of tasks, yet succeeded every time on only 13.41%. These results come from defined simulated tasks; real services still need separate verification of database conditions and side effects.
Back to Tech
ThinkingBox evaluates agent completion through database state and repeated runs
Single-attempt success differs from consistent reliability, and a normal tool call does not prove that the task was completed.