TY - GEN
T1 - A Study on Reproducibility and Replicability of Table Structure Recognition Methods
AU - Ajayi, Kehinde
AU - Choudhury, Muntabir Hasan
AU - Rajtmajer, Sarah M.
AU - Wu, Jian
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
PY - 2023
Y1 - 2023
N2 - Concerns about reproducibility in artificial intelligence (AI) have emerged, as researchers have reported unsuccessful attempts to directly reproduce published findings in the field. Replicability, the ability to affirm a finding using the same procedures on new data, has not been well studied. In this paper, we examine both reproducibility and replicability of a corpus of 16 papers on table structure recognition (TSR), an AI task aimed at identifying cell locations of tables in digital documents. We attempt to reproduce published results using codes and datasets provided by the original authors. We then examine replicability using a dataset similar to the original as well as a new dataset, GenTSR, consisting of 386 annotated tables extracted from scientific papers. Out of 16 papers studied, we reproduce results consistent with the original in only four. Two of the four papers are identified as replicable using the similar dataset under certain IoU values. No paper is identified as replicable using the new dataset. We offer observations on the causes of irreproducibility and irreplicability. All code and data are available on Codeocean at https://codeocean.com/capsule/6680116/tree.
AB - Concerns about reproducibility in artificial intelligence (AI) have emerged, as researchers have reported unsuccessful attempts to directly reproduce published findings in the field. Replicability, the ability to affirm a finding using the same procedures on new data, has not been well studied. In this paper, we examine both reproducibility and replicability of a corpus of 16 papers on table structure recognition (TSR), an AI task aimed at identifying cell locations of tables in digital documents. We attempt to reproduce published results using codes and datasets provided by the original authors. We then examine replicability using a dataset similar to the original as well as a new dataset, GenTSR, consisting of 386 annotated tables extracted from scientific papers. Out of 16 papers studied, we reproduce results consistent with the original in only four. Two of the four papers are identified as replicable using the similar dataset under certain IoU values. No paper is identified as replicable using the new dataset. We offer observations on the causes of irreproducibility and irreplicability. All code and data are available on Codeocean at https://codeocean.com/capsule/6680116/tree.
UR - http://www.scopus.com/inward/record.url?scp=85173587820&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85173587820&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-41679-8_1
DO - 10.1007/978-3-031-41679-8_1
M3 - Conference contribution
AN - SCOPUS:85173587820
SN - 9783031416781
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 3
EP - 19
BT - Document Analysis and Recognition – ICDAR 2023 - 17th International Conference, Proceedings
A2 - Fink, Gernot A.
A2 - Jain, Rajiv
A2 - Kise, Koichi
A2 - Zanibbi, Richard
PB - Springer Science and Business Media Deutschland GmbH
T2 - 17th International Conference on Document Analysis and Recognition, ICDAR 2023
Y2 - 21 August 2023 through 26 August 2023
ER -