TY - GEN
T1 - Online hate ratings vary by extremes
T2 - 4th ACM SIGIR Conference on Information Interaction and Retrieval, CHIIR 2019
AU - Salminen, Joni
AU - Almerekhi, Hind
AU - Kamel, Ahmed Mohamed
AU - Jung, Soon Gyo
AU - Jansen, Bernard J.
N1 - Publisher Copyright:
© 2019 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
PY - 2019/3/8
Y1 - 2019/3/8
N2 - Analyzing 5,665 crowd ratings on 1,133 social media comments, we find that individuals tend to agree on the extremes of a hate rating scale more than in the middle when evaluating the hatefulness of online comments. The agreement is higher for less hateful comments and lowest on moderately hateful comments. The results have implications for researchers developing machine learning models for online hate processing, as the extreme classes are likely to require fewer annotations for reaching statistical stability. Our findings suggest that the models developed in this domain should consider the distributions of hate ratings rather than average hate scores.
AB - Analyzing 5,665 crowd ratings on 1,133 social media comments, we find that individuals tend to agree on the extremes of a hate rating scale more than in the middle when evaluating the hatefulness of online comments. The agreement is higher for less hateful comments and lowest on moderately hateful comments. The results have implications for researchers developing machine learning models for online hate processing, as the extreme classes are likely to require fewer annotations for reaching statistical stability. Our findings suggest that the models developed in this domain should consider the distributions of hate ratings rather than average hate scores.
UR - http://www.scopus.com/inward/record.url?scp=85063134329&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85063134329&partnerID=8YFLogxK
U2 - 10.1145/3295750.3298954
DO - 10.1145/3295750.3298954
M3 - Conference contribution
AN - SCOPUS:85063134329
T3 - CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval
SP - 213
EP - 217
BT - CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval
PB - Association for Computing Machinery, Inc
Y2 - 10 March 2019 through 14 March 2019
ER -