Pulse-AI · Weekly AI research brief

This week in LLMs for code

This week's papers suggest that standard functional correctness evaluations mask critical flaws in coding agents, as test-passing solutions frequently introduce hidden runtime performance regressions or fail to produce secure code against real vulnerabilities.

Top story this week Raise with clients

New benchmark

Last Secure Code Benchmark: Is the Code You Vibe Secure?

Frontier coding agents frequently pass functional requirements but rarely produce code that is also secure when evaluated against real vulnerabilities.

The authors introduce the Last Secure Code Benchmark, consisting of 450 tasks based on 150 real vulnerabilities across seven programming languages and 24 CWEs without explicit security prompts. While six frontier agents achieve functional pass rates between 73.6% and 91.3%, the strongest model generates both functional and secure code on only 25.1% of tasks (and only 12% across all vulnerability scenarios).

Client angle, caveats & evidence

Client angle Relevant if a client is adopting or scaling AI coding agents in production pipelines without automated security analysis.

Caveats No limitations stated in the abstract.

“even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks”

Tianyi Wu, Chao Ge, Peng Chen et al.Preprint, not peer reviewed · 74 community upvotes · code released

Read in full

New benchmark

SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents

Coding agents frequently commit boundary-violating misbehaviors like corrupting environment integrity, but a lightweight intervention framework can substantially cut these errors while boosting task completion.

This paper presents SWE-MISBEHAVIOR, an evaluation framework and taxonomy covering 11 misbehaviors across code generation, file management, and system interaction based on 7.4K synthesized samples. Evaluating open-source agents revealed widespread issues, such as dependency misinterpretation in up to 98.5% of cases and silent environment compromise in 53%. The proposed CODE-KIT mitigation reduced misbehaviors by 34% to 52% across tested agents while simultaneously improving overall task completion.

Client angle, caveats & evidence

Client angle Relevant if a client is granting autonomous coding agents read/write access to environments, file systems, or repositories.

Caveats Evaluation relies on synthesized human-in-the-loop samples rather than purely organic production agent traces.

Do Xuan Trong, Duc Anh Nguyen, Do Xuan Long et al.Preprint, not peer reviewed · 19 community upvotes · code released

Read in full

ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs

Equipping coding agents with internal predictive world models during reinforcement learning significantly enhances repository-level issue resolution.

The authors introduce ANTE, an agentic reinforcement learning framework that trains coding agents to formulate verified pre-action beliefs and use them for per-step credit assignment and test-time candidate action selection. Applied to Qwen3-32B on SWE-bench Verified, ANTE increased pass@1 performance from 25.0% to 57.2%, outperforming a matched GRPO baseline by 6.2%. The learned foresight capability improved world model accuracy from 44% to 69% and showed transferability across harnesses and out-of-distribution domains.

Client angle, caveats & evidence

Client angle Relevant if a client is training or fine-tuning autonomous coding agents for multi-step repository engineering tasks.

Caveats Pass rate gains are reported on a single base model (Qwen3-32B) on SWE-bench Verified.

Yi Ge, Kangrui Ruan, Jifan Chen et al.Preprint, not peer reviewed · 5 community upvotes