Long-form question answering is one of those benchmarks that looks impressive on the leaderboard and disturbing under a microscope. This paper, called out in Jeff Dean's 2021 Google research retrospective, dissects the c-REALM system on ELI5-style questions and finds that many so-called improvements are artifacts of evaluation, retrieval noise, and dataset overlap rather than real reasoning gains.

The long review unpacks the specific hurdles: automatic metrics that reward fluent hallucination, retrieval components that barely influence the generated answer, and training-test contamination that inflates scores. For voice AI teams building conversational agents or RAG-powered voice assistants where users ask messy, open-ended questions, the takeaways matter directly, because a TTS voice reading a plausible-but-wrong long-form answer is uniquely dangerous. Give it a slot on your reading list if you're benchmarking retrieval-augmented QA and want to avoid publishing numbers you'll later have to walk back.