What Makes A Good Reference Manager? A Quantitative Analysis of Bibliography Management Applications

Tongan Cai, Chacha Chen, Ting Hao Huang, Frank E. Ritter

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations


Many researchers and students use reference managers to collect, manage, and format references and citations. While prior work has assessed these tools qualitatively, it is still unclear how to quantitatively evaluate reference managers. This paper starts to quantify the user effort required to use reference managers. We first collected surveys from 69 graduate students to understand their experience with reference managers, and then conducted user studies with 12 participants. In our study, each participant was asked to perform a standardized task using four popular reference managers: Mendeley, Zotero, EndNote, and RefWorks. We used RUI, a keystroke and mouse-move logger, to record the participants' activities and approximate their physical and mental effort. We also used pre-and post-study surveys to collect users' feedback and self-reported task load (as expressed by the NASA TLX Index.) The results showed that different reference managers require different levels of effort, and users generally prefer the tools that involve less effort. We also found that although reference managers share similar features, differences in presentation and organization matter. We conclude this work by providing a set of guidelines for both users and developers of reference managers.

Original languageEnglish (US)
Title of host publication5th Asian CHI Symposium 2021
EditorsJosh B. Tedjasaputra, Briane Paul V. Samson, Masitah Ghazali, Eunice Sari, Eunice Sari, Sayan Sarcar, Dilrukshi Gamage, Yohannes Kurniawan
PublisherAssociation for Computing Machinery, Inc
Number of pages6
ISBN (Electronic)9781450382038
StatePublished - May 8 2021
Event5th Asian CHI Symposium 2021 - Virtual, Online, Japan
Duration: May 7 2021May 8 2021

Publication series

Name5th Asian CHI Symposium 2021


Conference5th Asian CHI Symposium 2021
CityVirtual, Online

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Human-Computer Interaction
  • Information Systems
  • Software

Cite this