TY - GEN
T1 - ENHANCING PERCEPTION
T2 - 2024 Findings of the Association for Computational Linguistics: NAACL 2024
AU - Hsu, Yi Li
AU - Chen, Jui Ning
AU - Liu, Shang Chien
AU - Chiang, Yang Fan
AU - Xiong, Aiping
AU - Ku, Lun Wei
N1 - Publisher Copyright:
© 2024 Association for Computational Linguistics.
PY - 2024
Y1 - 2024
N2 - We introduce ENHANCING PERCEPTION, a framework for Large Language Models (LLMs) designed to streamline the time-intensive task typically undertaken by professional fact-checkers of crafting explanations for fake news. This study investigates the effectiveness of enhancing LLM explanations through conversational refinement. We compare various questioner agents, including state-of-the-art LLMs like GPT-4, Claude 2, PaLM 2, and 193 American participants acting as human questioners. Based on the histories of these refinement conversations, we further generate comprehensive summary explanations. We evaluated the effectiveness of these initial, refined, and summary explanations across 40 news claims by involving 2, 797 American participants, measuring their self-reported belief change regarding both real and fake claims after receiving the explanations. Our findings reveal that, in the context of fake news, explanations that have undergone conversational refinement-whether by GPT-4 or human questioners, who ask more diverse and detail-oriented questions-were significantly more effective than both the initial unrefined explanations and the summary explanations. Moreover, these refined explanations achieved a level of effectiveness comparable to that of expert-written explanations. The results highlight the potential of automatic explanation refinement by LLMs in debunking fake news claims.
AB - We introduce ENHANCING PERCEPTION, a framework for Large Language Models (LLMs) designed to streamline the time-intensive task typically undertaken by professional fact-checkers of crafting explanations for fake news. This study investigates the effectiveness of enhancing LLM explanations through conversational refinement. We compare various questioner agents, including state-of-the-art LLMs like GPT-4, Claude 2, PaLM 2, and 193 American participants acting as human questioners. Based on the histories of these refinement conversations, we further generate comprehensive summary explanations. We evaluated the effectiveness of these initial, refined, and summary explanations across 40 news claims by involving 2, 797 American participants, measuring their self-reported belief change regarding both real and fake claims after receiving the explanations. Our findings reveal that, in the context of fake news, explanations that have undergone conversational refinement-whether by GPT-4 or human questioners, who ask more diverse and detail-oriented questions-were significantly more effective than both the initial unrefined explanations and the summary explanations. Moreover, these refined explanations achieved a level of effectiveness comparable to that of expert-written explanations. The results highlight the potential of automatic explanation refinement by LLMs in debunking fake news claims.
UR - http://www.scopus.com/inward/record.url?scp=85197936880&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85197936880&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85197936880
T3 - Findings of the Association for Computational Linguistics: NAACL 2024 - Findings
SP - 2129
EP - 2147
BT - Findings of the Association for Computational Linguistics
A2 - Duh, Kevin
A2 - Gomez, Helena
A2 - Bethard, Steven
PB - Association for Computational Linguistics (ACL)
Y2 - 16 June 2024 through 21 June 2024
ER -