27 papers matching “benchmark”
Last Secure Code Benchmark: Is the Code You Vibe Secure?
Frontier coding agents frequently pass functional requirements but rarely produce code that is also secure when evaluated against real vulnerabilities.
- New benchmark
CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline
Multi-agent computer-use workflows suffer severe reliability breakdowns during conflict resolution, with frontier models failing more than half of basic coordination tasks such as releasing shared locks.
- New benchmark
SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents
Coding agents frequently commit boundary-violating misbehaviors like corrupting environment integrity, but a lightweight intervention framework can substantially cut these errors while boosting task completion.
- New benchmark
A Real-World CAD Segmentation Benchmark: Comparing Rule-Based and Learning-Based Approaches through Hole Detection
It introduces a human-annotated engineering CAD benchmark for hole detection and reveals that learning-based methods degrade under domain shift while rule-based alternatives maintain robust generalization.
- New benchmark
DecepEval: A Benchmark for Evaluating Deception in LLM Agents
Evaluations across nine frontier LLMs reveal that external inducements like pressure, incentive, opportunity, and conflict consistently increase agent deception across tasks, even for models with low baseline deception rates.
- New benchmark
Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents
It demonstrates that even when coding agents generate functionally correct patches, they violate repository-specific contribution and governance policies 43.1% of the time.
- New benchmark
- Surprising result
- Reframes the problem
Event Cameras for Melt-Pool Monitoring in Additive Manufacturing: A Benchmark and a Cross-Machine Transfer Analysis
Event-based monitoring matches dense-frame accuracy while providing a ~380x reduction in sensor readout and enabling spiking models with 15x lower modeled inference energy.
- New benchmark
Detecting Defects that Matter: An Application-Driven Benchmark for Anomaly Detection in Manufacturing and Retail Logistics (VAND 4.0 Challenge)
- New benchmark
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
A large-scale evaluation demonstrates that complex self-distillation methods in agentic RL do not reliably outperform standard GRPO, with extra compute failing to guarantee better results.
- New benchmark
- Surprising result
FDE-Gym: The Last Mile of AI Delivery
Frontier models fail dramatically at real-world customer AI delivery tasks, successfully resolving only 14 of 95 scenarios when required to uncover unstated business constraints.
- New benchmark
- Reframes the problem
MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments
Reinforcement learning trained on mocked GUI applications can scale to thousands of parallel instances and successfully transfer to physical devices, nearly doubling real-phone success rates.
- New benchmark
CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
Current coding agents achieve only a 32.30% mean Pass@1 when tasked with the end-to-end synthesis and refinement of reusable static-analysis checkers across real-world repositories.
- New benchmark
Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict
Higher agent task accuracy does not correlate with epistemic humility, as high-performing models often detect knowledge conflicts during execution but fail to escalate or report unresolved uncertainty in incorrect answers.
- New benchmark
- Surprising result
Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
Retaining raw action and feedback trajectories enables better in-context agent learning in novel environments than summarizing experiences into rules, though agents still lag behind human exploration capabilities.
- New benchmark
RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees
It improves agentic RAG accuracy on large enterprise documentation by dynamically constructing multi-document sub-trees from initial retrieval results to guide LLM exploration.
- New benchmark
Code Understanding is a Bottleneck for Coding Agents
It identifies that deep code understanding, rather than edit size, acts as the primary difficulty bottleneck and error driver for software engineering agents.
- New benchmark
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Equipping the generating LLM with workspace tools and verification skills enables autonomous verification and revision of long-horizon tasks without reference answers or rubrics.
- New benchmark
CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering
A unified benchmark evaluates agents on realistic software engineering workflows requiring concurrent code editing and graphical user interface interaction.
- New benchmark
Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering
- New benchmark
QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code
It demonstrates that domain-specific code specialization can significantly boost task and backtest success while simultaneously triggering capability-retention failures in general structured tool calling.
- New benchmark
Robust Logical Anomaly Detection under Capture Shifts via Synthetic Anomalies
It demonstrates that leading normal-reference industrial anomaly detectors suffer severe performance drops under simple capture condition shifts and provides a synthetic anomaly framework that retains high AUROC under perturbations.
- Surprising result
Inverting Multi-Vector Visual Document Indices
It demonstrates that multi-vector visual document indices stored in vector databases can be inverted to reconstruct document images, exposing 45% of sensitive tokens.
- Surprising result
UNREAL: Unifying Retrieval and Long-Context with a Single Model
It introduces a single model-internal selection mechanism adding under 500K parameters to a frozen LLM that improves both multi-hop corpus retrieval recall and long-context inference speed.
Sharpening Tax in Post-Training
RL post-training of LLM agents improves single-shot accuracy but systematically reduces solution coverage under repeated sampling.
- Surprising result
AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation
It introduces an evidence-first RAG release framework that reduces unsafe production deployments by treating LLM judges as fallible and quantifying regression risk probabilistically.
- Reframes the problem
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
It isolates the concrete performance impact of specific agent wrapper components, proving that harness design choice can add over 7 percentage points of performance independent of the base LLM.
MCFR: A Mask-Guided Coarse-to-Fine Regression Framework for Robust Multi-Variant Board-to-Board Connector Assembly
It demonstrates an average real-world insertion success rate of 99.25% for multi-variant board-to-board connectors in smartphone assembly by combining mask priors with photometric refinement.