A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing

Ting Hao Huang, Jeffrey P. Bigham

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Scopus citations

Abstract

A number of interactive crowd-powered systems have been developed to solve difficult problems out of reach for automated solutions. To work interactively, such systems need access to on-demand labor. To meet this demand, workers can be (i) recruited when needed directly from the crowd marketplace, or (ii) recruited in advance and asked to wait in a retainer pool until they are needed. Most of the evaluations of these systems have been over a short time period, even though we know that marketplaces change and adapt over time. In this paper, we present the results of a 10-month deployment of a crowd-powered system that uses a hybrid approach to fast recruitment of workers that we call Ignition. We describe the Ignition approach and the observed times required to recruit workers from the marketplace and retainer over this long period of time. Our results demonstrate that it is possible to recruit workers with low latency even over long periods, and suggest a number of opportunities for future work for recruitment strategies and modeling that may further improve ondemand recruitment for deployed systems.

Original languageEnglish (US)
Title of host publicationProceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017
EditorsSteven Dow, Adam Tauman
PublisherAAAI press
Pages61-70
Number of pages10
ISBN (Electronic)9781577357933
StatePublished - Oct 27 2017
Event5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 - Quebec City, Canada
Duration: Oct 24 2017Oct 26 2017

Publication series

NameProceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

Conference

Conference5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017
Country/TerritoryCanada
CityQuebec City
Period10/24/1710/26/17

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Human-Computer Interaction

Cite this