Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches

Rupika Dikkala; Roli Khanna; Caleb Matthews; Jonathan Dodge; Sai Raja; Catherine Hu; Jed Irvine; Zeyad Shureih; Kin Ho Lam; Andrew Anderson; Minsuk Kahng; Alan Fern; Margaret Burnett

doi:10.1109/CHASE52884.2021.00022

Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches

Rupika Dikkala, Roli Khanna, Caleb Matthews, Jonathan Dodge, Sai Raja, Catherine Hu, Jed Irvine, Zeyad Shureih, Kin Ho Lam, Andrew Anderson, Minsuk Kahng, Alan Fern, Margaret Burnett

College of Information Sciences and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Scopus citations

Abstract

How should empirical researchers conduct controlled, remote 'lab' studies in the uncontrolled, noisy conditions of each participant's own home? Volatility in participant home environments, hardware, internet connection, and surrounding distractions takes the 'controlled' out of controlled studies. This paper recounts our in-The-Trenches mitigations for designing and conducting two complex controlled studies under COVID, in which participants, from home, interactively localized faults in an AI system. The studies with our COVID-era mitigations in 5 categories-Privacy/Security, Data Collection, Control, Technology Issues, Payment-ultimately produced crisp results beyond what we thought possible under such uncontrolled circumstances.

Original language	English (US)
Title of host publication	Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	113-116
Number of pages	4
ISBN (Electronic)	9781665414098
DOIs	https://doi.org/10.1109/CHASE52884.2021.00022
State	Published - May 2021
Event	13th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021 - Virtual, Online Duration: May 20 2021 → May 21 2021

Publication series

Name	Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021

Conference

Conference	13th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021
City	Virtual, Online
Period	5/20/21 → 5/21/21

All Science Journal Classification (ASJC) codes

Software
Organizational Behavior and Human Resource Management
Communication

Access to Document

10.1109/CHASE52884.2021.00022

Cite this

Dikkala, R., Khanna, R., Matthews, C., Dodge, J., Raja, S., Hu, C., Irvine, J., Shureih, Z., Lam, K. H., Anderson, A., Kahng, M., Fern, A., & Burnett, M. (2021). Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches. In Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021 (pp. 113-116). Article 9463290 (Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CHASE52884.2021.00022

Dikkala, Rupika ; Khanna, Roli ; Matthews, Caleb et al. / Doing Remote Controlled Studies with Humans : Tales from the COVID Trenches. Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 113-116 (Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021).

@inproceedings{08c11ab6acab441dbc1a9d03121d6e7b,

title = "Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches",

abstract = "How should empirical researchers conduct controlled, remote 'lab' studies in the uncontrolled, noisy conditions of each participant's own home? Volatility in participant home environments, hardware, internet connection, and surrounding distractions takes the 'controlled' out of controlled studies. This paper recounts our in-The-Trenches mitigations for designing and conducting two complex controlled studies under COVID, in which participants, from home, interactively localized faults in an AI system. The studies with our COVID-era mitigations in 5 categories-Privacy/Security, Data Collection, Control, Technology Issues, Payment-ultimately produced crisp results beyond what we thought possible under such uncontrolled circumstances.",

author = "Rupika Dikkala and Roli Khanna and Caleb Matthews and Jonathan Dodge and Sai Raja and Catherine Hu and Jed Irvine and Zeyad Shureih and Lam, {Kin Ho} and Andrew Anderson and Minsuk Kahng and Alan Fern and Margaret Burnett",

note = "Funding Information: ACKNOWLEDGMENTS This work was supported in part by DARPA N66001-17-2-4030, NSF 1901031, and NSF 2042324. Publisher Copyright: {\textcopyright} 2021 IEEE.; 13th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021 ; Conference date: 20-05-2021 Through 21-05-2021",

year = "2021",

month = may,

doi = "10.1109/CHASE52884.2021.00022",

language = "English (US)",

series = "Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "113--116",

booktitle = "Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021",

address = "United States",

}

Dikkala, R, Khanna, R, Matthews, C, Dodge, J, Raja, S, Hu, C, Irvine, J, Shureih, Z, Lam, KH, Anderson, A, Kahng, M, Fern, A & Burnett, M 2021, Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches. in Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021., 9463290, Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021, Institute of Electrical and Electronics Engineers Inc., pp. 113-116, 13th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021, Virtual, Online, 5/20/21. https://doi.org/10.1109/CHASE52884.2021.00022

Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches. / Dikkala, Rupika; Khanna, Roli; Matthews, Caleb et al.
Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021. Institute of Electrical and Electronics Engineers Inc., 2021. p. 113-116 9463290 (Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Doing Remote Controlled Studies with Humans

T2 - 13th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021

AU - Dikkala, Rupika

AU - Khanna, Roli

AU - Matthews, Caleb

AU - Dodge, Jonathan

AU - Raja, Sai

AU - Hu, Catherine

AU - Irvine, Jed

AU - Shureih, Zeyad

AU - Lam, Kin Ho

AU - Anderson, Andrew

AU - Kahng, Minsuk

AU - Fern, Alan

AU - Burnett, Margaret

PY - 2021/5

Y1 - 2021/5

N2 - How should empirical researchers conduct controlled, remote 'lab' studies in the uncontrolled, noisy conditions of each participant's own home? Volatility in participant home environments, hardware, internet connection, and surrounding distractions takes the 'controlled' out of controlled studies. This paper recounts our in-The-Trenches mitigations for designing and conducting two complex controlled studies under COVID, in which participants, from home, interactively localized faults in an AI system. The studies with our COVID-era mitigations in 5 categories-Privacy/Security, Data Collection, Control, Technology Issues, Payment-ultimately produced crisp results beyond what we thought possible under such uncontrolled circumstances.

AB - How should empirical researchers conduct controlled, remote 'lab' studies in the uncontrolled, noisy conditions of each participant's own home? Volatility in participant home environments, hardware, internet connection, and surrounding distractions takes the 'controlled' out of controlled studies. This paper recounts our in-The-Trenches mitigations for designing and conducting two complex controlled studies under COVID, in which participants, from home, interactively localized faults in an AI system. The studies with our COVID-era mitigations in 5 categories-Privacy/Security, Data Collection, Control, Technology Issues, Payment-ultimately produced crisp results beyond what we thought possible under such uncontrolled circumstances.

UR - http://www.scopus.com/inward/record.url?scp=85113590503&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85113590503&partnerID=8YFLogxK

U2 - 10.1109/CHASE52884.2021.00022

DO - 10.1109/CHASE52884.2021.00022

M3 - Conference contribution

AN - SCOPUS:85113590503

T3 - Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021

SP - 113

EP - 116

BT - Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 20 May 2021 through 21 May 2021

ER -

Dikkala R, Khanna R, Matthews C, Dodge J, Raja S, Hu C et al. Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches. In Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 113-116. 9463290. (Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021). doi: 10.1109/CHASE52884.2021.00022

Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this