TY - GEN
T1 - DOCMATH-EVAL
T2 - 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
AU - Zhao, Yilun
AU - Long, Yitao
AU - Liu, Hongjun
AU - Kamoi, Ryo
AU - Nan, Linyong
AU - Chen, Lyuhao
AU - Liu, Yixin
AU - Tang, Xiangru
AU - Zhang, Rui
AU - Cohan, Arman
N1 - Publisher Copyright:
© 2024 Association for Computational Linguistics.
PY - 2024
Y1 - 2024
N2 - Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in real-world scenarios, particularly in expert domains, is still largely unexplored. This paper introduces DOCMATH-EVAL, a comprehensive benchmark specifically designed to evaluate the numerical reasoning capabilities of LLMs in the context of understanding and analyzing financial documents containing both text and tables. We evaluate a wide spectrum of 27 LLMs, including those specialized in math, coding and finance, with Chain-of-Thought and Program-of-Thought prompting methods. We found that even the current best-performing system (i.e., GPT-4) still significantly lags behind human experts in solving complex numerical reasoning problems grounded in long contexts. We believe DOCMATH-EVAL can be used as a valuable benchmark to evaluate LLMs' capabilities to solve challenging numerical reasoning problems in expert domains.
AB - Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in real-world scenarios, particularly in expert domains, is still largely unexplored. This paper introduces DOCMATH-EVAL, a comprehensive benchmark specifically designed to evaluate the numerical reasoning capabilities of LLMs in the context of understanding and analyzing financial documents containing both text and tables. We evaluate a wide spectrum of 27 LLMs, including those specialized in math, coding and finance, with Chain-of-Thought and Program-of-Thought prompting methods. We found that even the current best-performing system (i.e., GPT-4) still significantly lags behind human experts in solving complex numerical reasoning problems grounded in long contexts. We believe DOCMATH-EVAL can be used as a valuable benchmark to evaluate LLMs' capabilities to solve challenging numerical reasoning problems in expert domains.
UR - https://www.scopus.com/pages/publications/85204480650
UR - https://www.scopus.com/pages/publications/85204480650#tab=citedBy
U2 - 10.18653/v1/2024.acl-long.852
DO - 10.18653/v1/2024.acl-long.852
M3 - Conference contribution
AN - SCOPUS:85204480650
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 16103
EP - 16120
BT - Long Papers
A2 - Ku, Lun-Wei
A2 - Martins, Andre F. T.
A2 - Srikumar, Vivek
PB - Association for Computational Linguistics (ACL)
Y2 - 11 August 2024 through 16 August 2024
ER -