Collinear AI releases CWE-bench v2 with 100 public cybersecurity tasks across 91 CWEs and a leaderboard of 11 models

Collinear AI released CWE-bench v2, a public set of 100 vulnerability-fixing tasks covering 91 weakness types (CWEs). Each task is graded by code rather than by a person. It is a gated Hugging Face dataset, runnable through a tool called Harbor, and a separate held-out set stays private. Collinear's own leaderboard of 11 models has Gemini 4 Argon first at 67.8% pass@1 (81.0% pass@4), just ahead of Claude Opus 5.5 at 67.3% (78.0%). DeepSeek V4.1 Flash scores 57.0% at about $0.04 per attempt.

Security and AI teams get a cheap, reproducible way to compare how well models fix known vulnerabilities, and the cost-per-attempt figures help anyone weighing cheaper models. The tasks are also tagged for reinforcement-learning use, so labs could train defensive coding agents on them. The impact is modest: this is a second version of a benchmark from a smaller lab, not a new model or a rule change, and it is useful mainly if other groups adopt it.

The claim that it "tests the limits of frontier cyber capabilities" is promotional. A 0.5-point gap between the top two models, from one run of the lab's own harness, is within likely noise and says nothing about real-world security skill. Fixing a known, isolated bug in a graded task is also much easier than finding or patching flaws in messy production code. The 100 public tasks may leak into future training data, which would inflate scores over time.

1. 01Independent replication of the leaderboard by third parties, and whether the 67.8% versus 67.3% ordering holds across repeated runs
2. 02Whether model labs or other benchmark groups cite or adopt CWE-bench v2 in model cards and evaluations over the next few months
3. 03How the private held-out set compares with the public one: a large score gap would suggest contamination or overfitting to the public tasks