TY - GEN
T1 - CrowdEval
T2 - 17th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2018
AU - Qiu, Chenxi
AU - Squicciarini, Anna
AU - Khare, Dev Rishi
AU - Carminati, Barbara
AU - Caverlee, James
N1 - Publisher Copyright:
© 2018 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved.
PY - 2018
Y1 - 2018
N2 - Crowdsourcing platforms depend on the quality of work provided by a distributed workforce. Yet, it is challenging to dependably measure the reliability of these workers, particularly in the face of strategic or malicious behavior. In this paper, we present a dynamic and efficient solution to keep tracking workers' reliability. In particular, we use both gold standard evaluation and peer consistency evaluation to measure each worker performance, and adjust the proportion of the two types of evaluation according to the estimated distribution of workers' behavior (e.g., being reliable or malicious). Through experiments over real Amazon Mechanical Turk traces, we find that our approach has a significant gain in terms of accuracy and cost compared to state-of-the-art algorithms.
AB - Crowdsourcing platforms depend on the quality of work provided by a distributed workforce. Yet, it is challenging to dependably measure the reliability of these workers, particularly in the face of strategic or malicious behavior. In this paper, we present a dynamic and efficient solution to keep tracking workers' reliability. In particular, we use both gold standard evaluation and peer consistency evaluation to measure each worker performance, and adjust the proportion of the two types of evaluation according to the estimated distribution of workers' behavior (e.g., being reliable or malicious). Through experiments over real Amazon Mechanical Turk traces, we find that our approach has a significant gain in terms of accuracy and cost compared to state-of-the-art algorithms.
UR - http://www.scopus.com/inward/record.url?scp=85054646998&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85054646998&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85054646998
SN - 9781510868083
T3 - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
SP - 1486
EP - 1494
BT - 17th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2018
PB - International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)
Y2 - 10 July 2018 through 15 July 2018
ER -