视觉最佳实践
使用 Claude 视觉能力的最佳实践指南
🌐 查看英文原文 | 源码 Notebook
使用 Claude 视觉能力的最佳实践
视觉功能提供了一种与 Claude 交互的新模式。我们整理了一些技巧，帮助你在图像处理上获得最佳性能。在此之前，让我们先设置运行 notebook 所需的代码。
%pip install anthropic IPythonimport base64
from anthropic import Anthropic
from IPython.display import Image
client = Anthropic()
MODEL_NAME = "claude-opus-4-1"
def get_base64_encoded_image(image_path):
with open(image_path, "rb") as image_file:
binary_data = image_file.read()
base_64_encoded_data = base64.b64encode(binary_data)
base64_string = base_64_encoded_data.decode("utf-8")
return base64_string将传统技术应用于多模态
你可以使用角色设定等传统提示工程技术来修复幻觉问题。让我们看一个例子：
假设我想让 Claude 计算这张图片中狗的数量：
Image(filename="../images/best_practices/nine_dogs.jpg")<IPython.core.display.Image object>message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": get_base64_encoded_image("../images/best_practices/nine_dogs.jpg"),
},
},
{"type": "text", "text": "How many dogs are in this picture?"},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)The image shows a group of 10 dogs of various breeds sitting together in a grassy field with flowers in the background. The breeds appear to include Border Collies, an Australian Shepherd, and a Terrier mix, though I can't say for certain. The dogs have different colored coats including black, white, brown, and gray. They are all attentively facing the camera, likely posing for the photo.实际上只有 9 只狗，但 Claude 认为有 10 只！让我们应用一些提示工程再试一次。
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": get_base64_encoded_image("../images/best_practices/nine_dogs.jpg"),
},
},
{
"type": "text",
"text": "You have perfect vision and pay great attention to detail which makes you an expert at counting objects in images. How many dogs are in this picture? Before providing the answer in <answer> tags, think step by step in <thinking> tags and analyze every part of the image.",
},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)<thinking>
To accurately count the number of dogs in this image, I'll visually scan the image from left to right, focusing on each individual dog.
Starting on the left side, I see two black dogs standing next to each other. Moving to the right, there's a white and gray dog, then a brown and white dog, then a predominantly black dog with white markings.
Continuing to the right, I see a mostly black dog, then a black and white border collie type dog, then another dog with black and white fur.
Scanning the remaining right portion of the image, I see one more brown and white dog.
After carefully examining each dog in the photo, it appears I have accounted for all the dogs present.
</thinking>
<answer>There are 9 dogs total in this image of a diverse group of dogs sitting together in grass with some flowers in the background.</answer>太好了！在提示中应用了一些提示工程后，我们看到 Claude 现在正确地计数出有 9 只狗。
视觉提示
图像作为输入允许在图像本身中给出提示。让我们看一些例子。
在这张图片中，我们写了一些文字并画了一个箭头。让我们不带任何文本提示直接将其传递给 Claude。
Image(filename="../images/best_practices/circle.png")<IPython.core.display.Image object>message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/circle.png"),
},
},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)The image shows a simple black circle outline on a white background. Inside the circle, there is a straight horizontal line segment that does not touch the circle's edges. The circle and line are both drawn with thin, black strokes, giving the appearance of a basic geometric diagram or symbol.如你所见，因为我们没有给 Claude 提出问题，它试图描述图像。让我们在图像中添加一个问题，然后再传入一次。
Image(filename="../images/best_practices/labeled_circle.png")<IPython.core.display.Image object>message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/labeled_circle.png"),
},
},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)The area of the circle is πr^2, where r is the radius of the circle. The question states that the radius is 12, so the area would be π(12)^2 = 144π square units.我们还可以突出显示图像的特定部分并对此提出问题。
这两个数字之间的差异是多少？
Image(filename="../images/best_practices/table.png")<IPython.core.display.Image object>message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/table.png"),
},
},
{"type": "text", "text": "What's the difference between these two numbers?"},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)The difference between North America's net sales for the twelve months ended December 31, 2023 ($352,828) and December 31, 2022 ($315,880) is $36,948.少样本示例
在视觉任务中，向提示中添加示例同样可以提高准确性。让我们让 Claude 读取一张速度表的图片。
Image(filename="../images/best_practices/140.png")<IPython.core.display.Image object>message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/140.png"),
},
},
{"type": "text", "text": "What speed am I going?"},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)The speedometer in the image is showing a speed of 140 kilometers/hour (or about 87 miles/hour).Claude 的回答看起来不太对，它认为我们的速度是 140 公里/小时而不是 140 英里/小时！让我们再试一次，这次在提示中添加一些示例。
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/70.png"),
},
},
{"type": "text", "text": "What speed am I going?"},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "You are going 70 miles per hour."}],
},
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/100.png"),
},
},
{"type": "text", "text": "What speed am I going?"},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "You are going 100 miles per hour."}],
},
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/140.png"),
},
},
{"type": "text", "text": "What speed am I going?"},
],
},
]
response = client.messages.create(
model=MODEL_NAME, max_tokens=2048, messages=message_list, temperature=0
)
print(response.content[0].text)The speedometer in the image shows that you are going 140 miles per hour.完美！通过这些示例，Claude 学会了如何读取速度表上的速度。但请注意，图像的少样本提示并不总是有效，但值得在你的用例中尝试。
多图像输入
Claude 还可以一次接受并在提示中推理多个图像！例如，假设你有一张非常大的图像——比如一张很长的收据图像！我们可以将该图像分成多个块，然后将每个块输入给 Claude。
Image(filename="../images/best_practices/receipt1.png")<IPython.core.display.Image object>Image(filename="../images/best_practices/receipt2.png")<IPython.core.display.Image object>message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/receipt1.png"),
},
},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/receipt2.png"),
},
},
{"type": "text", "text": "Output the name of the restaurant and the total."},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)The name of the restaurant is The Breakfast Club and the total amount on the receipt is $78.86.从示例中识别物体
通过图像输入，你可以在提示中传入其他图像，Claude 将使用这些信息来回答问题。让我们看一个例子。
假设我们试图识别图像中裤子的类型。我们可以在提示中为 Claude 提供不同类型裤子的示例。
Image(filename="../images/best_practices/officer_example.png")<IPython.core.display.Image object>message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/wrinkle.png"),
},
},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/officer.png"),
},
},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/best_practices/chinos.png"),
},
},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image(
"../images/best_practices/officer_example.png"
),
},
},
{
"type": "text",
"text": "These pants are (in order) WRINKLE-RESISTANT DRESS PANT, ITALIAN MELTON OFFICER PANT, SLIM RAPID MOVEMENT CHINO. What pant is shown in the last image?",
},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)The last image shows a person wearing light gray wool dress pants or trousers paired with brown leather dress shoes or loafers. Based on the texture and drape of the fabric, these appear to be the Italian Melton Officer pants that were shown in the second product image.
Tip
中文读者提示：本章节的代码和输出为英文原文。如需理解具体实现细节，可参考上方中文导读和代码注释。如有疑问，欢迎在 GitHub Issues 讨论。