Retrieval-augmented generation

This week in Retrieval-augmented generation

This week's papers suggest that RAG bottlenecks often stem from factors beyond retriever inadequacy, such as search drift during multi-hop queries or generator visual constraints, meaning failures can persist even when retrieval succeeds.

Raise with clients

Reframes the problem

CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

It introduces CoverSynth, an automated framework that generates enterprise-grade retrieval evaluation sets from unannotated corpora to uncover failure modes and differentiate retrieval systems far more effectively than human-authored sets.

The authors introduce CoverSynth, a generator that creates synthetic retrieval evaluation sets designed for coverage, reliability, and discriminability without requiring expert manual labeling. Tested across five enterprise-style corpora including RepLiQA and TechQA, CoverSynth revealed overlooked failures like handling underspecified requests and yielded much more discriminative queries than human-generated sets. The paper also establishes an assessment framework to verify when evaluation sets satisfy these key testing criteria.

Client angle, caveats & evidence

Caveats Evaluation is demonstrated on public enterprise-style corpora rather than proprietary, live corporate deployment environments.

Anahita Sharma, Apaar Shanker, Rishav Chakravarti et al.Preprint, not peer reviewed

Read in full

Surprising result
Reframes the problem

Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG

The study demonstrates that apparent multi-hop retrieval failures often stem from search drift rather than retriever inadequacy, though recovering missing evidence primarily shifts errors to premature resolutions and abstentions.

Multi-hop retrieval-augmented generation (RAG) requires an agent to retrieve evidence for a sequence of dependent sub-questions. When a wrong answer lacks a required passage, the passage may be inaccessible to the retriever, or the agent may have failed to issue a query that would retrieve it.

Client angle, caveats & evidence

Client angle Relevant if a client is designing multi-hop RAG pipelines and trying to determine whether missing information is caused by poor retrieval models or query decomposition and evidence verification failures.

Caveats Results are reported exclusively on the MuSiQue dataset holding the generator and retriever fixed.

Jiaqi Wei, Wenxuan Huang, Chengdong Xu et al.Preprint, not peer reviewed