Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
In briefWhy search-augmented LLMs fail when reasoning over fast-changing knowledge
SummaryThe paper introduces TREK, a suite with TREKTRACE, a framework that attributes failures to specific retrieval or reasoning stages, plus two benchmarks: TREKBENCH-WILD for real-world questions and TREKBENCH-CTRL with controlled distractors. The authors find the main bottleneck is post-retrieval reasoning, as frontier models are often misled by conflicting evidence even after retrieving the needed information. They also show that benchmarks dominated by outdated distractors substantially underestimate task difficulty.