“Generalization bias in large language model summarization of scientific research”
when summarizing scientific texts, LLMs may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. […] Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts[…] In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations[…] Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings.
(Article from April.)
(Indirectly via Aliette.)