Pulse-AI · Weekly AI research brief

This week in LLMs for code

This week's papers suggest coding agents struggle with broader repository challenges: accuracy drops sharply on full repositories and complex code understanding, while agents violate repository policies 43.1% of the time even when patches are functionally correct.

Raise with clients

New benchmark
Surprising result
Reframes the problem

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

It demonstrates that even when coding agents generate functionally correct patches, they violate repository-specific contribution and governance policies 43.1% of the time.

The authors introduce SWE-CC, a benchmark with 823 machine-checkable policies across 12 open-source repositories to evaluate coding agent compliance during both runtime execution and final deliverables. Testing four LLMs across two agent scaffolds on 500 tasks revealed that modern agents violate 43.1% of applicable project policies despite passing functional tests. Nearly half of these policy violations occur during intermediate execution steps rather than just the final patch.

Client angle, caveats & evidence

Client angle Relevant if a client is considering deploying autonomous coding agents into production repositories and needs to ensure compliance with internal governance, testing workflows, and style standards beyond raw unit tests.

Caveats Evaluation is restricted to 12 open-source repositories and 500 tasks derived from SWE-bench Verified.

Hai Dang Truong, Rayner Goh, Thanh Le-Cong et al.Preprint, not peer reviewed

Read in full

New benchmark

Code Understanding is a Bottleneck for Coding Agents

It identifies that deep code understanding, rather than edit size, acts as the primary difficulty bottleneck and error driver for software engineering agents.

The researchers created CABRA, a synthetic diagnostic benchmark with 6,840 tasks built as call graph transformations to isolate specific agent abilities like function traversal and search. They observed that agents handle large tasks by offloading search to tools, and that tool call counts predict SWE-bench Verified accuracy better than lines of code edited. When subjected to tasks requiring analysis of divergent logic across classes, agent accuracy dropped significantly.

Client angle, caveats & evidence

Client angle Relevant if a client is trying to understand why coding agents fail on complex legacy codebases despite low edit volume.

Caveats The primary findings rely on synthetic call graph transformation tasks paired with an observational study on SWE-bench Verified.

Nishant Balepur, Kiran Tomlinson, Tobias SchnabelPreprint, not peer reviewed