Good Data, Large Data, or No Data? Comparing Three Approaches in Developing Research Aspect Classifiers for Biomedical Papers

Shreya Chandrasekhar, Chieh Yang Huang, Ting Hao Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

The rapid growth of scientific publications, particularly during the COVID-19 pandemic, emphasizes the need for tools to help researchers efficiently comprehend the latest advancements. One essential part of understanding scientific literature is research aspect classification, which categorizes sentences in abstracts to Background, Purpose, Method, and Finding. In this study, we investigate the impact of different datasets on model performance for the crowd-annotated CODA-19 research aspect classification task. Specifically, we explore the potential benefits of using the large, automatically curated PubMed 200K RCT dataset and evaluate the effectiveness of large language models (LLMs), such as LLaMA, GPT-3, ChatGPT, and GPT-4. Our results indicate that using the PubMed 200K RCT dataset does not improve performance for the CODA-19 task. We also observe that while GPT-4 performs well, it does not outperform the SciBERT model fine-tuned on the CODA-19 dataset, emphasizing the importance of a dedicated and task-aligned datasets dataset for the target task. Our code is available at https://github.com/Crowd-AI-Lab/CODA-19-exp.

Original languageEnglish (US)
Title of host publicationBioNLP 2023 - BioNLP and BioNLP-ST, Proceedings of the Workshop
EditorsDina Demner-fushman, Sophia Ananiadou, Kevin Cohen
PublisherAssociation for Computational Linguistics (ACL)
Pages103-113
Number of pages11
ISBN (Electronic)9781959429852
StatePublished - 2023
Event22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, BioNLP 2023 - Toronto, Canada
Duration: Jul 13 2023 → …

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, BioNLP 2023
Country/TerritoryCanada
CityToronto
Period7/13/23 → …

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Cite this