The Ecology of Harmful Design: Risk and Safety of Game Making on a Metaverse Platform

Metaverse platforms have been on the rise in recent years, offering three-dimensional (3D), immersive virtual worlds while encouraging user-generated content (UGC) in various forms. Roblox, a popular metaverse platform, enables its users to create a holistic virtual world (i.e., develop a 3D game) for other users to interact with. However, complex UGC is also challenging to moderate. Roblox has been notorious for its users’ harmful designs, such as Nazi or terrorist role-playing mechanisms. In this study, we explore how harmful design takes place on Roblox. Through a grounded theory analysis of the ‘r/Robloxgamedev’ subreddit, we conceptualize an ecological view of harmful design, foregrounding three interconnected circumstances, namely sociotechnical risks, socioeconomic precarities, and normative (in)sensitivities, which work together to condition and give rise to harmful designs and bring about unique governance challenges to metaverse platforms. We conclude by laying out implications for design moderation.