Early detection of policies violations in a social media site: A Bayesian belief network approach

One of the main goals of all online social communities is to promote a stable, or perhaps, growing membership built around topics of like interest. Yet, communities are not impermeable to the potentially damaging effects resulting from those few participants that choose to behave in a manner that is counter to established norms of behavior. Typical moderators in online social communities are the ones tasked to reduce the risks associated with unhealthy user behavior by rapidly identifying and removing damaging posts and consequently taking action against the perpetrating user. Yet, the sheer volume of posts relative to the number of moderators available for review suggests a need for modern tools aimed at prioritizing posts based on the assessed risk each user poses to the community. To accomplish this, we propose a threat analysis model. Our model, referred to as TrICO (Threat requires Intent Capability and Opportunity) is implemented using Bayesian Networks, and achieves early detection of damaging behavior in online social communities. To the best of our knowledge, this is the first user-centered model for usage policy enforcement in online sites. We apply our model to a comprehensive data set characterizing the entirety of a popular discussion forum. Our results show that the TrICO model provides accurate results.