Collinear AI released 100 CWE-bench v2 tasks that ask agents to find and fix weaknesses in real open-source repositories. They span 91 weakness categories and programmatically check that existing functionality survives. The reported leading model scored 67.8% pass@1 and 81.0% pass@4, leaving failures even after four attempts. Harbor environments and verifiers are available, with acceptance of dataset access conditions required. Scores describe success on specified tasks, not an automatic guarantee of security in a deployed service.
Back to Tech
CWE-bench v2 releases 100 tasks for evaluating security fixes
Deterministic checks also require preserved functionality, but strong scores do not guarantee every weakness is fixed.