Whether it be by posting a photo on Instagram or writing a blog post, we’re all adding more information to the internet. With over 4.62 billion people using social media, there are bound to be some bad eggs creating harmful or deceitful content. To make sure that users are exposed to as little bad content as possible, websites practice content moderation.

Content moderation is the process of regulating and monitoring user-generated content based on a set of pre-existing rules and guidelines. Since it is near impossible for human moderators to ever catch up to the huge amount of content being posted every second on social media, some websites are now using artificial intelligence (AI) to moderate posts. Let’s dive deeper into how AI content moderators work and whether they are actually effective in making the internet safer.

What is AI moderation?

AI moderators are fed user-generated content to train them on what is acceptable and what’s not on a particular platform. The AI will learn to pick up patterns and filter out illegal, dangerous or sexually explicit content. Automating the content moderation process with AI allows a large amount of user-generated content to be checked in real-time. Platforms can thus be proactive in taking down any suspicious content that might be harmful to their users. Using AI moderators not only reduces the workload on human moderators but also reduces the chances of them being exposed to offensive content which can adversely affect their mental health.

Two layers of content moderation

There are two broad layers of moderating content—surface-level moderation and context-based moderation. In the first layer, image, text and video content are checked through natural language processing to recognize damaging elements. If there is text inside an image, it can be further checked using object character recognition, a technology that converts words inside images into a format that the AI can understand. An example of an AI content moderator is Checkstep, which compares user-generated content against a company’s terms and conditions to check for violations.

The second layer requires the AI to understand the nuance in which a particular statement is being made. While this form of AI moderation is still in its nascent stages, tech company Spectrum Labs has created the AI moderator Contextual AI, which gives us some clues on what context-based moderation could look like. Contextual AI takes into account current and historical responses to a user’s content to understand the context in which it is being created and understood.

Is AI alone enough to monitor content?

Although AI moderation can help social media giants monitor content faster, it alone isn’t entirely effective. Since AI understands malign content based on the data it has been exposed to, it wouldn’t be able to flag harmful ideas that it doesn’t recognize. Moreover, even though there are attempts to contextualize content with technologies, it still has a long way to go before achieving accurate comprehension. At present, even if made in a completely harmless context, the use of some words, like “stupid”, might still end up giving the content a high toxicity score.

The issues with AI moderators have been well documented in the media. Recently, Daniel Motaung, a former content moderator at Facebook, shared that even though the social media giant claims that its AI has 95% accuracy in detecting hate speech, there are times when it removes content that doesn’t violate content policies. The same issue has been reported with other platforms too. For instance, in 2017 YouTube took down videos of journalists reporting on extremist content because its AI moderator didn’t know the difference between reportage and content inciting extremist behavior. This lack of detection and differentiation capabilities can wrongfully punish users, some of whom might be full-time content creators, and take down their content.

Finally, much like other kinds of AI, AI moderators are also susceptible to bias based on the data that they have been trained on. Much like the issues around Google’s allegedly sentient AI, LaMDA, the kind of content being fed to the AI moderator also raises serious ethical questions. Ultimately, even though AI can detect problematic text and imagery, human moderators, who are better equipped to read between the lines, are still essential in content moderation.

Header image courtesy of Freepik