AI agents · Pulse-AI

This week in AI agents

This week’s papers suggest frontier models face severe reliability bottlenecks in practical execution, struggling to manage multi-agent coordination conflicts in computer use and failing the vast majority of real-world customer delivery scenarios.

More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning

A large-scale evaluation demonstrates that complex self-distillation methods in agentic RL do not reliably outperform standard GRPO, with extra compute failing to guarantee better results.

The authors evaluated ten representative self-distillation agent reinforcement learning methods against GRPO across four environments, three model sizes, and 132 training runs. They found that no single self-distillation method consistently outperformed GRPO and that higher training costs did not reliably translate to superior results. Additionally, strong final task scores frequently concealed underlying training instability.

Client angle, caveats & evidence

Client angle Relevant if a client is considering investing substantial engineering time and GPU budgets into complex self-distillation algorithms for agent training rather than using standard reinforcement learning baselines.

Caveats Evaluation was conducted entirely on synthetic research benchmarks (ALFWorld, ScienceWorld, Search-QA, and WebShop) rather than production enterprise systems.

“gains over GRPO remain limited in scope, with no method consistently outperforming it across settings.”

Linhao Zhong, Bozhen Fang, Xuehao Wang et al.Preprint, not peer reviewed · 82 community upvotes · code released

Top story this week Raise with clients

New benchmark
Reframes the problem

FDE-Gym: The Last Mile of AI Delivery

Frontier models fail dramatically at real-world customer AI delivery tasks, successfully resolving only 14 of 95 scenarios when required to uncover unstated business constraints.

The authors developed FDE-Gym, a benchmark of 95 end-to-end delivery tasks based on real customer scenarios across 18 sectors to test agents on ambiguous requirements and live system interactions. Evaluating 13 configurations across seven providers showed that top models scored poorly, resolving only 14 tasks without incident. Even when agents successfully elicited every key fact from stakeholders, they still failed to achieve high business uplift on the majority of tasks.

Client angle, caveats & evidence

Client angle Relevant if a client is considering deploying autonomous agents to handle ambiguous enterprise integration or engineering tasks without human forward-deployed engineers.

Caveats Customer environments and stakeholder responses are simulated and scripted rather than conducted with actual live customer teams.

“the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks”

Dongxu Yang, Hongda Yan, Renxin Wang et al.Preprint, not peer reviewed · 284 community upvotes · code released

Read in full

New benchmark

CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline

Multi-agent computer-use workflows suffer severe reliability breakdowns during conflict resolution, with frontier models failing more than half of basic coordination tasks such as releasing shared locks.

The paper introduces CLASHarness, a benchmark with 600 verified multi-round tasks designed to evaluate how vision-language agents manage conflicting instructions, stale state information, and shared file permissions. Testing 15 frontier models revealed that actively fixing system state is significantly harder for agents than merely detecting inconsistencies. Furthermore, coordination protocols proved to be a universal bottleneck, with frontier models failing over 58% of closing actions.

Client angle, caveats & evidence

Caveats The conflict scenarios and hybrid verification rubrics are programmatically synthesized rather than captured from real human-agent corporate IT deployments.

Yuxiang Lai, Jike Zhong, Chengzhi Liu et al.Preprint, not peer reviewed · 33 community upvotes · code released

Read in full

New benchmark

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

Reinforcement learning trained on mocked GUI applications can scale to thousands of parallel instances and successfully transfer to physical devices, nearly doubling real-phone success rates.

The researchers introduced MAI-Gym, an open platform scaling up to 20,000 parallel instances for GUI agent evaluation and online reinforcement learning across mobile, computer, and web environments. Using the platform's synthesized tasks and mocked environments, they trained a 9B model that reached state-of-the-art performance among sub-10B open models on MobileWorld and OSWorld-Verified. Crucially, RL training in a mocked application transferred directly to a physical smartphone, increasing success from 28.8% to 48.1%.

Client angle, caveats & evidence

Caveats Physical real-world transfer was only demonstrated on a single mobile train-ticketing application.

Panrong Tong, Quyu Kong, Pengxiang Li et al.Preprint, not peer reviewed · 154 community upvotes · code released