Introduction
Content Moderator work is critical to the safety of online platform users and the integrity of these platforms or search engines. This type of work has many benefits, yet also carries specific risks. Most user-generated reviews by moderators are compliant, conform to community standards, and can even be monotonous or humorous. However, some can have unexpected or surprising impacts. In addition, their work can be cognitively complex; juggling pressures of high-performance standards, time demands, supporting frustrated account holders, and keeping on top of policy changes, all while at risk of exposure to sometimes graphic, disturbing, and shocking content. Without sufficient support, repeated and/or overexposure to this type of content increases the risk of psychological and emotional distress.
Impacts
The specific aspects of viewing media content that contribute to traumatic responses are unexpected and surprising content, repeated exposure, and duration of exposure. Also, personal connection with content (affiliated with a community, group, or nationality) or feelings of guilt and shame (relating to feeling unaffected) can increase impact. Moderators can experience a range of emotional responses, such as disgust, fear, shock, anger, and sadness. Physical symptoms can include body tension, heart palpitations, poor concentration, perspiration, and headaches3 as content may activate sympathetic nervous system responses.
Startle Response Habituation
However, some studies have shown that reoccurring exposure to surprising or shocking content eventually leads to habituation of responses and reduced psychophysiological reactions. Essentially, moderators get used to provoking content over time, particularly when working on the same category of queues or type of content. A recent qualitative study observed the first signs of nervous system activation in content moderators between training and production, when content volume increases, and they are now required to adhere to specific performance metrics. Cognitive load increases, however protective factors such as finding a sense of purpose help moderators cope and adapt. Viewing themselves as protectors against societal harm online and bringing justice to those afflicted, the study found, helps moderators adjust to content.
However, this study also found that certain aspects of content moderator workflow interfere with adaptation to content. Stress and cognitive overload from keeping up with changing policies and changing protocols deteriorated their ability to process graphic content. This is an important factor to consider from an organizational context, to ensure that work processes minimize stress and cognitive overload to help manage any unnecessary impact. Moderators in this study also reported that wellbeing supports such as psychoeducational training, and skill-building group and individual sessions helped them to process content and manage impacts, even calling for more wellbeing initiatives to support them in their roles. Another reported protective factor identified in this study related to transparency during the recruitment process. Moderators reported that being fully informed about the nature of the role with detailed examples of content increased preparedness for the role and helped manage the impact of first exposure to content.
However, taking into consideration the findings in this study, it is important to consider whether reduced psychophysiological impact over time is an adaptive protective factor for moderators. From a trauma-informed perspective, is it an adaptive process of habituation or a form of dissociation of experience that may have longer-term consequences? Exploring this and other cognitive emotional impacts of viewing content should be explored in future research.
Post-traumatic Resilience
Sometimes trauma survivors or those in occupations at risk of vicarious trauma (therapists, journalists, and Content Moderators) can in fact experience positive changes after adverse experiences. Alongside the potential negative impact of exposure to trauma, some may have a new sense of strength, purpose, and need to help others; to make meaning out of experiences that may be difficult to make sense of. Post-traumatic growth is a construct used to describe this form of transformation that occurs following adversity and psychological struggle. In addition to this, a qualitative study found that certain factors, such as social and organizational support predicted growth. Therefore, it is so important to ensure that Content Moderators have a supportive social network in a nurturing workplace environment to allow for this form of resilience and adaptation to develop.
Conclusion
Many factors influence the impact of viewing graphic or disturbing content on moderators’ cognitive and emotional wellbeing. Adhering to queues with similar content, reducing duration and exposure, particularly with highly egregious content, and reducing unnecessary, additional cognitive load or stress can help reduce startle response activation. Also ensuring there are established colleague and management support systems and wider organizational support provide a resource and opportunity for growth if moderators feel they have been impacted.
