Watch Your Language: Investigating Content Moderation with Large Language Models

Content moderation has always been a job for large teams of human reviewers. Right after ChatGPT came out, the obvious question was whether a model could just do it instead.

So we measured it. We tested GPT-3, GPT-3.5, GPT-4, Gemini Pro, and LLAMA 2 on two jobs a real moderator does: enforcing one community’s specific rules, and catching toxic content in general.

For the first, we built 95 subreddit-specific moderators by prompting GPT-3.5 with each community’s actual rules, then gave them real comments from that community. For the second, we compared the models against commercial toxicity classifiers on a balanced set of 10,000 comments.

Enforcing a community’s rules

Median accuracy across 95 subreddits

64%

Median precision

83%

Best community (r/movies)

82% accuracy, 95% precision

When GPT-3.5 flagged a comment, it was usually right. It also missed a lot, though: with a median accuracy of 64%, roughly a third of violations got through.

Results also varied a lot from one community to the next. r/movies came close to what human moderators do. Communities whose rules depend on conversational context or on things nobody wrote down did much worse, which makes sense, since that is exactly the part the model has to guess.

The most practical finding is that the model did better with restrictive rules (don’t do X) than with prescriptive ones (be welcoming, contribute constructively). If you’re writing rules that an LLM will enforce, how you phrase them matters more than you’d expect.

Detecting toxicity

Off-the-shelf LLMs did better than the commercial toxicity classifiers people already pay for. GPT-3.5 was the most balanced, at 0.73 accuracy and 0.75 F1, while the baselines mostly landed between 0.60 and 0.65.

What this means in practice

For platforms, LLMs are useful for communities with clear, restrictive rules, as long as a human stays in the loop. A 64% median accuracy means about one in three violations still gets through.

For people writing community policy, rules phrased as restrictions hold up better than rules phrased as goals. Rules that depend on context will keep tripping up automated systems no matter which model is enforcing them.

Overall, LLMs are a real step forward for moderation, but they aren’t ready to replace human judgment yet.