The increasing sophistication of AI in content moderation

Recent research from the Consumer Technology Association shows that user-generated content (UGC) now accounts for 39% of the time Americans spend consuming media every week, compared with 61% for traditional media. Whether they’re watching TikTok videos, following gamers on Twitch or leaving product reviews, consumers are spending more time than ever engaging with UGC.

According to cloud software company DOMO, every one minute of the day in 2021, users shared 65,000 photos on Instagram, sent 668,000 messages via Discord, streamed 694,000 hours on YouTube and watched 44 million Facebook Live videos. The total amount of data consumed globally in 2021 was 79 zettabytes, per Statista. To put this into context, one zettabyte is equal to one trillion gigabytes, and incredibly, this number, and the amount of content posted to social media, community forums and other sites is projected to grow to over 180 zettabytes by 2025.

What does all of that mean for brands? More UGC means there’s more data to monitor in order to ensure you aren’t inadvertently aligning yourself with inappropriate, violent or fake content. Additionally, the more UGC there is, the higher the chances are that bad actors will post something objectionable — making content moderation a colossal task that simply cannot be achieved efficiently without the support of artificial intelligence (AI).

The battle for control

The problem is that while AI is becoming more sophisticated, the individuals creating negative digital content are, too. “The challenge is that it’s an arms race,” Nigel Duffy, AI entrepreneur and global AI leader at Ernst & Young, explains. “The sophistication of the tools to generate content is competing against the capacity to moderate that content—and right now, the former is winning.”

Duffy predicts that we’re about to experience a “tsunami of content moderation.” As brands fight to keep up with those creating negative content online, AI will become even more essential to their success.

Content moderation best practices: Maintaining brand trust in a digital world

Learn the importance of content moderation, how to choose the right method and the various ways to measure your success.

Download the report

The benefits of AI in content moderation

Because of the necessity and the sheer volume and variety of content, companies are already finding ways to capitalize on AI technology to protect their brands and maintain a positive customer experience. Naturally, speed and scalability are among the features that give it so much appeal. With artificial intelligence, brands can process large amounts of content in very little time — something that simply can’t be matched by manual content moderation by humans alone.

While image content and text data can contain content that’s harmful to both consumers and moderators, tools like natural language processing (NLP), image processing algorithms, sentiment analysis and computer vision can all help brands defend against violence, harassment and more.

Introducing a human touch

But that isn’t to say AI-enabled content moderation is without its limitations. Brands can learn how to train AI systems to spot harmful images — but human oversight helps determine the context of potentially harmful text, and human-led data annotation ensures the placements of metadata tags to the original dataset, which provides a layer of richer information to support machine learning.

“It’s useful to have humans-in-the-loop,” Duffy says. “If you think about graphic violence in a video game or a screen capture versus in real life, it can be relatively hard for AI to distinguish sometimes.” The same, he notes, is true of what people define as appropriate or inappropriate levels of violence. “There are boundaries,” Duffy continues, “and AI and humans need to work in tandem to identify them.”

Because it’s critical for businesses to protect the mental health and overall wellness of their customer care teams, they’re leveraging automated content filtering to identify banned behavior and content. Better tagging weeds out the most flagrant instances of violence and labels the content so that moderators have an idea of what they are about to view. Moreover, research shows that interactive image blurring can reduce the emotional impact and overall strain on content moderators “without sacrificing accuracy or speed.” Viewing content in grayscale can also have a positive effect on the moderators, while still enabling them to flag violent and extreme material.

Reducing human exposure to harmful content helps to mitigate the psychological, emotional and physical impact of toxic content on human moderators. This is even true of content that’s being delivered in real-time, as AI can moderate livestreams and automatically delete harmful material before it’s posted. Ideally, humans play a key role, but AI should be the first line of defense.

Avoiding AI bias

Organizations should also strive to hire and maintain diverse and inclusive teams throughout the data collection and labeling process. Assembling a diverse team of data annotators and validators can help brands make the right judgment calls and help reduce bias in AI. Because this process starts with people, it can be a slippery slope for homogenous groups to let subtle, unconscious biases enter the algorithms without a second-opinion and rigorous testing backup. As such, diverse teams help reduce bias because they make room for new perspectives, backgrounds and experiences that might not have been considered and fed into the AI systems initially.

For example, a recent study found that AI models trained by African-American users to process hate speech online were 1.5 times more likely to identify tweets as offensive or hateful. This is important because the feedback these human moderators provide is incorporated back into the AI training loop, which in turn helps brands train their AI systems responsibly. If the team isn’t diverse, neither is the AI in its thinking.

AI is only getting smarter, with technological capabilities that range from pattern recognition to reading language context. Regardless, humans will likely always be better at identifying the emotion and intent behind digital content. The solution? Humans and AI must work together to monitor content more effectively and create better and more inclusive experiences for all.

This article is part of a five-part series on content moderation. Check out our other articles on the evolving nature of digital content, wellness strategies for content moderators, content moderation regulations and what's next for content moderation.