Pulse-AI · Weekly AI research brief

This week in LLMs for code

This week's papers suggest that while structured tool use provides stable gains for software engineering agents, domain-specific code specialization can simultaneously degrade general structured tool calling capabilities.

Skim it

New benchmark

QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code

It demonstrates that domain-specific code specialization can significantly boost task and backtest success while simultaneously triggering capability-retention failures in general structured tool calling.

The authors examine specializing LLMs for Backtrader algorithmic trading code through continued pretraining and agent-validated supervised fine-tuning (SFT). Evaluating on their 400-task QuantCode-Bench and a repository-level setup, SFT following pretraining raised Qwen3.6-35B-A3B agentic success from 47.5% to 79.5% after repair. However, domain specialization degraded structured tool calling, which was not fully recovered in repository-level agent performance by targeted recovery SFT.

Client angle, caveats & evidence

Client angle Relevant if a client is considering fine-tuning or continually pretraining open-weight coding models on proprietary, domain-specific programming frameworks.

Caveats Evaluation is centered on a single domain framework (Backtrader) and the authors' own new benchmark.

Alexey Chernysh, Orkhan Ekhtibarov, Dmitry ZmitrovichPreprint, not peer reviewed · 20 community upvotes

Read in full

Beyond the Model: Demystifying Harness Effects in Software Engineering Agents

It isolates the concrete performance impact of specific agent wrapper components, proving that harness design choice can add over 7 percentage points of performance independent of the base LLM.

This empirical study investigates the impact of agent harness components on software engineering performance using ten open-weight models across three benchmarks. The authors find that harness utility depends heavily on task complexity and base model strength, showing that complex harnesses offer diminishing returns on standard issue repair. They also develop NanoHarness, showing that structured tool use and task-specific subagents provide stable gains whereas context compression and general subagents can harm generation.

Client angle, caveats & evidence

Client angle Relevant if a client is architecting custom agent frameworks or scaffolds for coding workflows rather than relying solely on frontier foundation model upgrades.

Caveats The empirical testing is limited to open-weight models from the Qwen and DeepSeek families across three specific benchmarks.

Haichuan Hu, Quanjun Zhang, Shengcheng Yu et al.Preprint, not peer reviewed