In Cognition’s FrontierCode 2.0 evaluation, the best model solved 45.5% of tasks that included hidden requirements. Its 132 real-repository tasks test not only requested features but also implicit constraints such as surrounding code and compatibility. Cognition says the benchmark expands repository coverage, blockers and the scope of changes compared with its predecessor. More reasoning can also produce unnecessary changes. Evaluating coding agents therefore calls for checking requirement coverage and change scope separately from test success.
Back to Tech
Coding agents solve just 45.5% of tasks with hidden requirements
Passing tests does not necessarily mean completing a real repository task