IProWA: A Novel Probabilistic Graphical Model for Crowdsourcing Aggregation

Crowdsourcing has become a popular way to obtain a large volume of labeled data due to its low cost and high efficiency. Usually the crowdsourcing process enables redundancy in the collected labels in order to ensure the correctness of item labels. However, workers on the crowdsourcing platform may make mistakes on some items, leading to inconsistent labels. In this case, it is important to aggregate these noisy labels and obtain the true labels of the items. The correctness of the item label provided by a worker depends on both the worker's ability and the property of the item. However, most of the existing models consider the effect of workers' abilities but ignore that of the item properties. In this paper, we propose a novel crowdsourcing aggregation method (IProWA) which incorporates the modeling of not only worker expertise level but also item property. In particular, items are represented by a K dimensional vector (i.e., item parameter), where K is the number of possible categories and each dimension represents a category. The proposed model transforms the true label estimation into the estimation of item parameters as it connects the true label and the parameters of an item. In worker modeling, it models the different category propensities among different worker groups. Experimental results show that the performance of the proposed model is comparable to that of the state-of-the-art baselines and the learned item parameters can help interpret the property of that item.