构建内容审核过滤器
用 Claude 创建内容安全过滤
🌐 查看英文原文 | 源码 Notebook
使用 Claude 构建内容审核过滤器
本指南将向你展示如何使用 Claude 为用户生成的文本构建内容审核过滤器。核心思想是将审核规则和类别直接定义在 prompt 中，从而便于自定义和实验。
基本方法
基本方法是向 Claude 提供一个 prompt，描述你想要过滤的类别（例如”ALLOW”和”BLOCK”），以及每个类别应该包含哪些类型内容的详细描述或示例。然后，你将需要分类的用户生成文本插入到 prompt 中，并要求 Claude 根据提供的指南对其进行分类。
以下是一个 prompt 结构示例：
You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines:
BLOCK CATEGORY:
- [Description or examples of content that should be blocked]
ALLOW CATEGORY:
- [Description or examples of content that is allowed]
Here is the user-generated text to categorize:
<user_text>{{USER_TEXT}}</user_text>
Based on the guidelines above, classify this text as either ALLOW or BLOCK. Return nothing else.使用时，你需要将 {USER_TEXT} 替换为需要分类的实际用户生成文本，然后使用 Claude API 将 prompt 发送给 Claude。Claude 的回复应该是”ALLOW”或”BLOCK”，表示根据你提供的指南应如何处理该文本。
使用示例
以下是一些演示如何使用此方法的 Python 示例代码：
%pip install anthropicfrom anthropic import Anthropic
client = Anthropic()
MODEL_NAME = "claude-haiku-4-5"
def moderate_text(user_text, guidelines):
prompt_template = """
You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines:
{guidelines}
Here is the user-generated text to categorize:
<user_text>{user_text}</user_text>
Based on the guidelines above, classify this text as either ALLOW or BLOCK. Return nothing else.
"""
# Format the prompt with the user text
prompt = prompt_template.format(user_text=user_text, guidelines=guidelines)
# Send the prompt to Claude and get the response
response = (
client.messages.create(
model=MODEL_NAME, max_tokens=10, messages=[{"role": "user", "content": prompt}]
)
.content[0]
.text
)
return response下面是一个示例，展示如何使用此函数来审核一组用户评论：
example_guidelines = """BLOCK CATEGORY:
- Promoting violence, illegal activities, or hate speech
- Explicit sexual content
- Harmful misinformation or conspiracy theories
ALLOW CATEGORY:
- Most other content is allowed, as long as it is not explicitly disallowed
"""
user_comments = [
"This movie was great, I really enjoyed it. The main actor really killed it!",
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Thanks for the helpful information!",
]
for comment in user_comments:
classification = moderate_text(comment, example_guidelines)
print(f"Comment: {comment}\nClassification: {classification}\n")Comment: This movie was great, I really enjoyed it. The main actor really killed it!
Classification: ALLOW
Comment: Delete this post now or you better hide. I am coming after you and your family.
Classification: BLOCK
Comment: Stay away from the 5G cellphones!! They are using 5G to control you.
Classification: BLOCK
Comment: Thanks for the helpful information!
Classification: ALLOW自定义
此方法的一个关键好处是，你可以通过修改 prompt 中”BLOCK”和”ALLOW”类别的描述或示例来轻松自定义审核规则。这使你能够微调过滤行为以适应特定的需求或偏好。
例如，如果你想让 Claude 审核一个过山车爱好者论坛，确保帖子保持在主题范围内，你可以相应地更新”ALLOW”和”BLOCK”类别的描述：
rollercoaster_guidelines = """BLOCK CATEGORY:
- Content that is not related to rollercoasters, theme parks, or the amusement industry
- Explicit violence, hate speech, or illegal activities
- Spam, advertisements, or self-promotion
ALLOW CATEGORY:
- Discussions about rollercoaster designs, ride experiences, and park reviews
- Sharing news, rumors, or updates about new rollercoaster projects
- Respectful debates about the best rollercoasters, parks, or ride manufacturers
- Some mild profanity or crude language, as long as it is not directed at individuals
"""
post_titles = [
"Top 10 Wildest Inversions on Steel Coasters",
"My Review of the New RMC Raptor Coaster at Cedar Point",
"Best Places to Buy Cheap Hiking Gear",
"Rumor: Is Six Flags Planning a Giga Coaster for 2025?",
"My Thoughts on the Latest Marvel Movie",
]
for title in post_titles:
classification = moderate_text(title, rollercoaster_guidelines)
print(f"Title: {title}\nClassification: {classification}\n")Title: Top 10 Wildest Inversions on Steel Coasters
Classification: ALLOW
Title: My Review of the New RMC Raptor Coaster at Cedar Point
Classification: ALLOW
Title: Best Places to Buy Cheap Hiking Gear
Classification: BLOCK
Title: Rumor: Is Six Flags Planning a Giga Coaster for 2025?
Classification: ALLOW
Title: My Thoughts on the Latest Marvel Movie
Classification: BLOCK通过思维链（CoT）提升性能
一种可以增强 Claude 内容审核能力的技术是”思维链”（Chain of Thought, CoT）prompting。这种方法鼓励 Claude 将其推理过程分解为逐步的思维链，而不仅仅是提供最终输出。
要在审核中利用思维链，你可以修改 prompt，明确指示 Claude 在 <thinking> 标签内将其过程分解为清晰的步骤。以下是一个示例：
cot_prompt = """You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines:
BLOCK CATEGORY:
- Content that is not related to rollercoasters, theme parks, or the amusement industry
- Explicit violence, hate speech, or illegal activities
- Spam, advertisements, or self-promotion
ALLOW CATEGORY:
- Discussions about rollercoaster designs, ride experiences, and park reviews
- Sharing news, rumors, or updates about new rollercoaster projects
- Respectful debates about the best rollercoasters, parks, or ride manufacturers
- Some mild profanity or crude language, as long as it is not directed at individuals
First, inside of <thinking> tags, identify any potentially concerning aspects of the post based on the guidelines below and consider whether those aspects are serious enough to block the post or not. Finally, classify this text as either ALLOW or BLOCK inside <output> tags. Return nothing else.
Given those instructions, here is the post to categorize:
<user_post>{user_post}</user_post>"""
user_post = "Introducing my new band - Coaster Shredders. Check us out on YouTube!!"
response = (
client.messages.create(
model=MODEL_NAME,
max_tokens=1000,
messages=[{"role": "user", "content": cot_prompt.format(user_post=user_post)}],
)
.content[0]
.text
)
print(response)<thinking>
The post appears to be promoting a band rather than discussing rollercoasters, theme parks, or the amusement industry. This falls under the "spam, advertisements, or self-promotion" category, which is grounds for blocking the post.
</thinking>
<output>BLOCK</output>通过示例提升性能
另一种提升性能的方法是在 prompt 中添加少量示例，为 Claude 提供一些初始训练数据或”少样本学习”（few-shot learning），使其更好地理解所需的分类方式。这对于细微差别或模糊的情况特别有帮助，因为在这些情况下，仅从文本描述可能无法完全清楚地判断类别边界。以下是如何修改 prompt 模板以包含示例的示例：
examples_prompt = """You are a content moderation expert tasked with categorizing user-generated text based on the following guidelines:
BLOCK CATEGORY:
- Content that is not related to rollercoasters, theme parks, or the amusement industry
- Explicit violence, hate speech, or illegal activities
- Spam, advertisements, or self-promotion
ALLOW CATEGORY:
- Discussions about rollercoaster designs, ride experiences, and park reviews
- Sharing news, rumors, or updates about new rollercoaster projects
- Respectful debates about the best rollercoasters, parks, or ride manufacturers
- Some mild profanity or crude language, as long as it is not directed at individuals
Here are some examples:
<examples>
Text: I'm selling weight loss products, check my link to buy!
Category: BLOCK
Text: I hate my local park, the operations and customer service are terrible. I wish that place would just burn down.
Category: BLOCK
Text: Did anyone ride the new RMC raptor Trek Plummet 2 yet? I've heard it's insane!
Category: ALLOW
Text: Hercs > B&Ms. That's just facts, no cap! Arrow > Intamin for classic woodies too.
Category: ALLOW
</examples>
Given those examples, here is the user-generated text to categorize:
<user_text>{user_text}</user_text>
Based on the guidelines above, classify this text as either ALLOW or BLOCK. Return nothing else."""
user_post = "Why Boomerang Coasters Ain't It (Don't @ Me)"
response = (
client.messages.create(
model=MODEL_NAME,
max_tokens=1000,
messages=[{"role": "user", "content": examples_prompt.format(user_text=user_post)}],
)
.content[0]
.text
)
print(response)ALLOW
Tip
中文读者提示：本章节的代码和输出为英文原文。如需理解具体实现细节，可参考上方中文导读和代码注释。如有疑问，欢迎在 GitHub Issues 讨论。