CrowDXR - Pitfalls and potentials of experiments with remote participants

Jiayan Zhao, Mark Simpson, Pejman Sajjadi, Jan Oliver Wallgrün, Ping Li, Mahda M. Bagher, Danielle Oprean, Lace Padilla, Alexander Klippel

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations

Abstract

Although the COVID-19 pandemic has made the need for remote data collection more apparent than ever, progress has been slow in the virtual reality (VR) research community, and little is known about the quality of the data acquired from crowdsourced participants who own a head-mounted display (HMD), which we call crowdXR. To investigate this problem, we report on a VR spatial cognition experiment that was conducted both in-lab and out-of-lab. The in-lab study was administered as a traditional experiment with undergraduate students and dedicated VR equipment. The out-of-lab study was carried out remotely by recruiting HMD owners from VR-related research mailing lists, VR subreddits in Reddit, and crowdsourcing platforms. Demographic comparisons show that our out-of-lab sample was older, included more males, and had a higher sense of direction than our in-lab sample. The results of the involved spatial memory tasks indicate that the reliability of the data from out-of-lab participants was as good as or better than their in-lab counterparts. Additionally, the data for testing our research hypotheses were comparable between in- and out-of-lab studies. We conclude that crowdsourcing is a feasible and effective alternative to the use of university participant pools for collecting survey and performance data for VR research, despite potential design issues that may affect the generalizability of study results. We discuss the implications and future directions of running VR studies outside the laboratory and provide a set of practical recommendations.

Original languageEnglish (US)
Title of host publicationProceedings - 2021 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021
EditorsMaud Marchal, Jonathan Ventura, Anne-Helene Olivier, Lili Wang, Rafael Radkowski
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages450-459
Number of pages10
ISBN (Electronic)9781665401586
DOIs
StatePublished - 2021
Event20th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021 - Virtual, Online, Italy
Duration: Oct 4 2021Oct 8 2021

Publication series

NameProceedings - 2021 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021

Conference

Conference20th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021
Country/TerritoryItaly
CityVirtual, Online
Period10/4/2110/8/21

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'CrowDXR - Pitfalls and potentials of experiments with remote participants'. Together they form a unique fingerprint.

Cite this