裁剪放大工具
给 Claude 一个裁剪放大工具以读取图像细节
🌐 查看英文原文 | 源码 Notebook
为 Claude 提供放大工具以读取图像细节
简介
当图像中的细节太小而无法辨认时，你会凑近仔细看。Claude 自身无法做到这一点：它以固定的有效分辨率查看图像一次，当图表标签或一对间距很近的线条在该分辨率下太小时，任何提示都无法恢复它。
放大工具赋予了 Claude 凑近观察的能力。
当 Claude 请求查看某个区域的特写时，该工具从全分辨率原始图像中裁剪该区域并返回放大后的结果，使小细节变得可读。本 notebook 构建了这个工具，在公开的图表阅读基准测试上，它的准确率提高了一倍以上（结果见下文）。为什么放大会有效，从 Claude 实际查看图像的方式来看，稍后会解释。
在本教程结束时，你将能够：
- 使用公开文档中的
resized_size()参考实现预调整图像大小，使 Claude 输出的像素坐标与你持有的图像 1:1 对应 - 定义一个自包含的放大工具——输入绝对像素坐标，输出填充预算的放大裁剪——你可以直接移植到任何项目中
- 运行围绕它的 Agent 循环，包括防止长且思考密集的回合被静默截断的停止原因处理
图表是本 notebook 的持续示例，但相同的模式适用于任何细节相对于图像较小的情况：密集文档、UI 截图、原理图、扫描表格、标签照片。
结果，先看
示例问题
在第 48 个月，哪个产品线的值更高：Widget-A 还是 Widget-E？ 在该月，两条线相距约 26 个单位——在此渲染中约为 8 像素——正好在一个真正的交叉点旁边。（这是本 notebook 为自己生成的演示问题，采用以下基准测试使用的密集真实世界图表的风格；基准测试自身的问题是留出的评估集，此处不复现。）
有工具和无工具的对比
在没有提示的情况下，模型回答 Widget-A——很自信，通常将附近的交叉描述为已经发生——在 40 次记录尝试中有 36 次是错误的（90%）。有了放大工具，它会反复放大交叉区域，每次裁剪都比上一次更紧凑，从放大视图中读取顺序，然后回答 Widget-E——正确，在 40 次记录尝试中 40 次正确（100%），使用的正是本 notebook 构建的代码。（这是这一个演示问题的重复运行；下面的基准测试数字是独立的单次测量。）你稍后将在 notebook 中亲自运行这个交互。
数字
同样的模式，在 Chartography 上测量——这是 Surge AI 的一个公开基准测试，包含 100 道关于密集真实世界图表的图表阅读题——每个模型对每道题提问两次（有工具和无工具），并根据数据集卡片用模型评判器对自由格式答案进行评分。下面的每个柱代表一个模型在一个条件下——所有 100 道题的准确率——箭头标记添加工具后的变化；每题的平均成本在图注中。
这些数字来自使用图表中标注的模型对公开 Claude API 运行基准测试的结果：每个模型每个条件下所有 100 道题的准确率（2026-07-18），成本为 30 题使用样本的平均值（2026-07-19），按公开 API 价格计算（Sonnet 5 按其发布价格），评判器未计入。放大分支使用提示缓存测量，无工具分支作为单次未缓存调用，因为缓存不适用于一次性请求；循环的总结调用不命中缓存，因此优化后的客户端运行成本会略低。
注意： Chartography 是一个留出的评估集——不要用于训练（数据集卡片正是为此目的提供了 Canary 字符串）。提示、黄金答案和元数据的版权归 Surge AI 所有，CC BY 4.0。来自网络的图表图像仍归其原始权利持有人所有；本教程不重新分发数据集中的任何内容。
前置条件
所需知识： - Python 基础 - 对 Messages API 中的工具调用有基本了解
所需工具： - Python 3.11 或更高版本 - Anthropic API 密钥（在此获取）
环境配置
%pip install -q anthropic pillow matplotlibNote: you may need to restart the kernel to use updated packages.import base64
from collections.abc import Iterable, Mapping
from io import BytesIO
from typing import Any, cast
from anthropic import Anthropic
from IPython.display import Image, display
from PIL import Image as PILImage
client = Anthropic()
MODEL = "claude-fable-5"创建演示图表
我们将绘制放大工具所针对的那种图表：六条系列、密集的网格，以及用故意微小的字号打印出我们问题的答案。自己生成意味着我们确切知道真实答案——而且注释在全视图中确实太小而无法阅读。
import matplotlib.pyplot as plt
import numpy as np
rng = np.random.default_rng(12)
months = np.arange(60)
fig, ax = plt.subplots(figsize=(16, 9), dpi=100)
series = {}
for name, base, amp in [
("Widget-A", 5200, 420),
("Widget-B", 4800, 380),
("Widget-C", 5600, 450),
("Widget-D", 4400, 300),
("Widget-E", 5014, 340),
("Widget-F", 4600, 410),
]:
y = base + amp * np.sin(months / 7 + base % 7) + rng.normal(0, 90, months.size).cumsum() / 4
series[name] = y
ax.plot(months, y, lw=1.0, label=name)
# The detail our question hinges on: a tiny annotation at Widget-C's peak
peak_x = int(np.argmax(series["Widget-C"]))
peak_y = float(series["Widget-C"][peak_x])
ax.scatter([peak_x], [peak_y], s=12, color="crimson", zorder=3)
ax.annotate(
f"peak: {peak_y:,.0f} units",
(peak_x, peak_y),
textcoords="offset points",
xytext=(5, 4),
fontsize=5,
)
ax.set_title("Monthly production by product line")
ax.set_xlabel("Month")
ax.set_ylabel("Units")
ax.grid(True, lw=0.3, alpha=0.6)
ax.legend(fontsize=7, ncols=3)
# Hand the figure to the rest of the notebook as a PIL image
buffer = BytesIO()
fig.savefig(buffer, format="PNG", bbox_inches="tight")
plt.close(fig)
buffer.seek(0)
chart_image = PILImage.open(buffer)
question = "What exact value is annotated at the peak of the Widget-C line?"
answer = f"peak: {peak_y:,.0f} units"
print(f"Chart size: {chart_image.size}")
print(f"Question: {question}")
_buf = BytesIO()
chart_image.save(_buf, format="PNG")
display(Image(data=_buf.getvalue(), width=560))Chart size: (1325, 777)
Question: What exact value is annotated at the peak of the Widget-C line?<IPython.core.display.Image object>Claude 如何查看图像——以及为什么使用像素坐标
关于Claude 如何处理图像的两个事实决定了工具的设计：
图像以视觉 token 为单位衡量。 一个图像每 28×28 像素的块消耗一个视觉 token，每个模型都有图像预算：在标准分辨率层级，任何一边不得超过 1568 像素，且一个图像最多消耗 1568 个视觉 token；高分辨率层级模型（如
claude-fable-5和claude-sonnet-5）允许 2576 像素和 4784 个视觉 token。超过限制的图像会在 Claude 查看之前自动缩小——小细节也随之缩小。Claude 最适合使用绝对像素坐标。 视觉坐标指南建议明确要求像素坐标，并建议不要使用归一化（0–1）坐标。有一个注意点：Claude 使用的坐标是Claude 看到的图像中的像素位置——在任何自动缩小之后。解决方法是在上传之前自行调整图像大小，使你持有的图像恰好是 Claude 看到的图像，每个坐标都 1:1 对应。
我们将裁剪图放大到图像预算允许的最大尺寸并返回。从原始图像而非缩小后的副本裁剪，意味着放大功能可以利用源图像的每个像素。
- 上传前： 调整我们展示给 Claude 的图像大小，使其发送给我们的放大坐标与我们持有的图像 1:1 对应。
- 放大时： 将裁剪区域放大到预算内的最大尺寸，使其中的每个元素获得尽可能多的块。
定义放大工具
每次放大调用的流程：
- Claude requests a region as pixel coordinates
(x1, y1)–(x2, y2), with the origin(0, 0)at the top-left corner (x increases to the right, y increases downward). - We map those coordinates onto the full-resolution original (which may be larger than the copy Claude saw) and crop the region from it.
- We scale the crop up to the largest size the image budget allows and return it.
Cropping from the original rather than from the downscaled copy means the zoom works with every pixel the source image has.
One practical detail: zoomed crops are returned as JPEG rather than PNG. Tool results accumulate in the conversation, and a few full-budget PNG crops of a detailed chart can exceed the API’s 32 MB request size limit. (For conversations that accumulate many large images, the Files API keeps request payloads small regardless of history length.)
The next cell is self-contained — schema and implementation, no other cells needed — so you can copy it directly into your own project.
# The zoom tool: schema + implementation. This cell is self-contained (it needs
# only Pillow, plus the anthropic SDK for its type annotations), so you can copy it
# straight into your own project.
import base64
import math
from io import BytesIO
from anthropic.types import ImageBlockParam, TextBlockParam, ToolParam
from PIL import Image as PILImage
# Image limits for the model you're using — see "Resolution and token cost":
# https://platform.claude.com/docs/en/build-with-claude/vision#evaluate-image-size
MAX_EDGE = 2576 # high-resolution tier; use 1568 for standard-tier models
MAX_TOKENS = 4784 # high-resolution tier; use 1568 for standard-tier models
def count_image_tokens(width: int, height: int) -> int:
"""Visual tokens consumed by an image: one token per 28x28 pixel patch."""
return math.ceil(width / 28) * math.ceil(height / 28)
# Reference implementation from the vision coordinates guide ("Resize your image
# before uploading"):
# https://platform.claude.com/docs/en/build-with-claude/vision-coordinates#resize-your-image-before-uploading
def resized_size(
width: int,
height: int,
max_edge: int = MAX_EDGE,
max_tokens: int = MAX_TOKENS,
) -> tuple[int, int]:
"""The size Claude resizes an image to before padding.
Images that already fit within the limits are returned unchanged.
"""
def fits(w: int, h: int) -> bool:
return (
math.ceil(w / 28) * 28 <= max_edge
and math.ceil(h / 28) * 28 <= max_edge
and count_image_tokens(w, h) <= max_tokens
)
if fits(width, height):
return (width, height)
if height > width:
resized_h, resized_w = resized_size(height, width, max_edge, max_tokens)
return (resized_w, resized_h)
# Binary search along the long edge for the largest aspect-preserving
# size that fits.
aspect_ratio = width / height
lo, hi = 1, width # lo always fits; hi never fits
while lo + 1 < hi:
mid = (lo + hi) // 2
if fits(mid, max(round(mid / aspect_ratio), 1)):
lo = mid
else:
hi = mid
return (lo, max(round(lo / aspect_ratio), 1))
def prepare_image(image: PILImage.Image) -> PILImage.Image:
"""Resize an image to exactly the size Claude will see.
The pixel coordinates Claude uses refer to the image it saw — after any
automatic server-side downscaling. Doing the same resize client-side first
means Claude's coordinates map 1:1 onto the image we hold.
"""
target = resized_size(image.width, image.height)
if (image.width, image.height) == target:
return image
return image.resize(target, PILImage.Resampling.LANCZOS)
def zoom_size(width: int, height: int) -> tuple[int, int]:
"""The largest aspect-preserving size within the image limits.
This is the upscale target for a zoomed crop: the larger the crop is drawn,
the more 28x28 patches each element inside it covers. resized_size() finds
the largest size that fits whenever its input is over the limits, so we
scale the crop's size far past the limits and let the search come back down.
"""
return resized_size(width * 10_000, height * 10_000)
def pil_to_base64(image: PILImage.Image, format: str = "PNG") -> str:
"""Convert PIL Image to base64 string.
Transparency is composited onto white: a bare .convert("RGB") would
flatten transparent backgrounds (common in web charts) to black.
"""
if image.mode != "RGB":
rgba = image.convert("RGBA")
background = PILImage.new("RGB", rgba.size, "white")
background.paste(rgba, mask=rgba.getchannel("A"))
image = background
buffer = BytesIO()
# subsampling=0 keeps full chroma resolution: 4:2:0 subsampling halves color
# detail, which can smear 1-2px colored chart lines in a magnified crop.
image.save(
buffer, format=format, **({"quality": 92, "subsampling": 0} if format == "JPEG" else {})
)
return base64.standard_b64encode(buffer.getvalue()).decode("utf-8")
# Tool definition for the Anthropic API
ZOOM_TOOL: ToolParam = {
"name": "zoom",
"description": (
"Zoom into a rectangular region of an image. Returns the region cropped "
"from the full-resolution original and magnified, so that small details "
"become legible."
),
"input_schema": {
"type": "object",
"properties": {
"x1": {
"type": "integer",
"description": "Left edge of the region, in pixels (the origin is the image's top-left corner)",
},
"y1": {
"type": "integer",
"description": "Top edge of the region, in pixels",
},
"x2": {
"type": "integer",
"description": "Right edge of the region, in pixels (must be greater than x1)",
},
"y2": {
"type": "integer",
"description": "Bottom edge of the region, in pixels (must be greater than y1)",
},
"image_index": {
"type": "integer",
"description": "Which image to zoom into, counting images in the conversation from 0. Omit when there is only one image.",
},
},
"required": ["x1", "y1", "x2", "y2"],
},
}
def handle_zoom(
original: PILImage.Image, view_size: tuple[int, int], x1: int, y1: int, x2: int, y2: int
) -> list[TextBlockParam | ImageBlockParam]:
"""Execute a zoom call and return the tool result content for Claude.
original is the full-resolution image; view_size is the size of the copy
Claude saw (the output of prepare_image()), which is the coordinate space
of x1/y1/x2/y2.
"""
view_w, view_h = view_size
x1, x2 = max(0, min(x1, view_w)), max(0, min(x2, view_w))
y1, y2 = max(0, min(y1, view_h)), max(0, min(y2, view_h))
if x1 >= x2 or y1 >= y2:
return [
{
"type": "text",
"text": "Error: invalid region (need x1 < x2 and y1 < y2, within the image)",
}
]
# Map view coordinates onto the original. The resize preserved the aspect
# ratio, so one scale factor covers both axes.
scale = original.width / view_w
box = (round(x1 * scale), round(y1 * scale), round(x2 * scale), round(y2 * scale))
cropped = original.crop(box)
zoomed = cropped.resize(zoom_size(cropped.width, cropped.height), PILImage.Resampling.LANCZOS)
# Zoomed crops go back as JPEG: every tool result stays in the conversation,
# and a few full-budget PNGs of detailed charts can exceed the API's 32 MB
# request size limit. JPEG keeps each zoom to a fraction of that.
return [
{
"type": "text",
"text": (
f"Zoomed into ({x1},{y1})-({x2},{y2}) of the image you see at "
f"{view_w}x{view_h}px: a {cropped.width}x{cropped.height}px region "
f"of the original, returned magnified to {zoomed.width}x{zoomed.height}px."
),
},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": pil_to_base64(zoomed, "JPEG"),
},
},
]在将放大工具交给 Claude 之前，让我们手动测试一下——放大到峰值标注所在的图表上部区域：
def show_tool_result(blocks: Iterable[Mapping[str, Any]], show_text: bool = True) -> None:
"""Print the text and display the images among a tool result's content blocks.
A notebook convenience for inspecting what a zoom call returns. It reads the
blocks loosely (they are plain dicts on their way to the API), so it is not
part of the tool itself.
"""
for block in blocks:
if block.get("type") == "text" and show_text:
print(block["text"])
elif block.get("type") == "image":
display(Image(data=base64.b64decode(block["source"]["data"]), width=400))
view = prepare_image(chart_image)
print(f"Original size: {chart_image.size}, size Claude sees: {view.size}")
# Zoom into the upper-middle band of the chart
result = handle_zoom(chart_image, view.size, x1=0, y1=0, x2=view.width, y2=int(view.height * 0.3))
show_tool_result(result)Original size: (1325, 777), size Claude sees: (1325, 777)
Zoomed into (0,0)-(1325,233) of the image you see at 1325x777px: a 1325x233px region of the original, returned magnified to 2576x453px.<IPython.core.display.Image object>Agent 循环
现在我们将一切连接起来：将图像连同可用的放大工具发送给 Claude，让 Claude 反复调用直到得出答案。SDK 的 tool_runner 驱动这个交换——每当模型请求时它调用我们的 zoom 函数，将放大的裁剪图反馈回去，并在模型回答时停止（或在达到 max_iterations 后停止，这是防止失控循环的保障）。
这里有三个细节很重要：
- 我们同时保留两者：全分辨率原始图像（供工具裁剪）和预调整大小的副本（Claude 实际看到的，其大小定义了坐标空间）。
zoom函数闭包捕获两者。 - 提示说明每个图像的尺寸和坐标约定，使 Claude 确切知道它在哪个像素空间中工作。
@beta_tool从函数的签名和文档字符串构建工具的模式——要添加第二个工具，装饰另一个函数并将其添加到tools列表中；运行器按名称分发。
一个棘手之处：API 可能会在服务器端暂停一个非常长的回合（stop_reason == "pause_turn"），而运行器本身尚未恢复这些——下面围绕它的小循环会在发生这种情况时重放对话记录并继续。
from anthropic import beta_tool
from anthropic.types.beta import BetaContentBlockParam, BetaMessageParam
def ask_with_zoom_tool(images, question: str, verbose: bool = True, model: str = MODEL):
"""Ask Claude a question about one or more images, with the zoom tool available."""
if not isinstance(images, list):
images = [images]
originals = images
views = [prepare_image(image) for image in originals]
@beta_tool
def zoom(x1: int, y1: int, x2: int, y2: int, image_index: int = 0) -> list:
"""Zoom into a rectangular region of an image to see it magnified.
Use this whenever text, numbers, lines, or details are too small to read
confidently in the full view. The region is cropped from the image's
full-resolution original and scaled up to fill the image budget, so
small elements become much easier to read. Coordinates are absolute
pixels in the image as you see it, origin at the top-left corner.
Zoom again on the result for even finer detail.
Args:
x1: Left edge of the region in pixels (origin is the image's top-left corner).
y1: Top edge of the region in pixels.
x2: Right edge of the region in pixels (must be greater than x1).
y2: Bottom edge of the region in pixels (must be greater than y1).
image_index: Which image to zoom into (0 for the first image, 1 for the second).
"""
if not 0 <= image_index < len(originals):
return [{"type": "text", "text": f"Error: image_index {image_index} is out of range"}]
return handle_zoom(originals[image_index], views[image_index].size, x1, y1, x2, y2)
content: list[BetaContentBlockParam] = []
for i, view in enumerate(views):
content.append({"type": "text", "text": f"Image {i} ({view.width}x{view.height} pixels):"})
content.append(
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": pil_to_base64(view),
},
}
)
content.append(
{
"type": "text",
"text": (
f"Answer the following question about the image{'s' if len(views) > 1 else ''} "
f"above. Coordinates are absolute pixels with the origin at the top-left corner. "
f"Use the zoom tool to examine any detail that is too small to read confidently."
f"\n\nThe question is: {question}"
),
}
)
messages: list[BetaMessageParam] = [{"role": "user", "content": content}]
runner = final = None # bound before the loop for the type checker; the body always runs
for _reentry in range(5): # pause_turn re-entries are bounded, like the tool loop itself
runner = client.beta.messages.tool_runner(
model=model,
max_tokens=8192,
tools=[zoom],
messages=messages,
max_iterations=20,
)
for message in runner:
called = [block for block in message.content if block.type == "tool_use"]
if verbose:
for block in called:
print(f"[Tool] {block.name}({block.input})")
if called:
# The runner caches this response and reuses it internally, so
# peeking at it here does not re-run the tool. It is the
# tool_result message the runner sends next; its blocks are the
# plain dicts handle_zoom() built, so read them as such.
peek = cast(Any, runner.generate_tool_call_response())
for tool_result in peek["content"] if peek else []:
# An errored tool call carries a plain string instead
# of blocks; only successful zooms have images to show.
if isinstance(tool_result["content"], list):
show_tool_result(tool_result["content"], show_text=False)
final = runner.until_done()
if final.stop_reason != "pause_turn":
break
# The server paused a long turn; the runner doesn't resume these itself,
# so replay the transcript so far and continue where it left off. The
# counter records how often that happened.
globals().setdefault("PAUSE_TURN_RESUMES", []).append(_reentry + 1)
snapshot = {}
runner.set_messages_params(lambda params, _out=snapshot: (_out.update(params), params)[1])
messages = [*snapshot["messages"], {"role": "assistant", "content": final.content}]
assert runner is not None and final is not None # the loop ran at least once
answer = "\n".join(block.text for block in final.content if block.type == "text")
if not answer.strip():
# The run ended without a text answer - usually the iteration budget ran
# out mid-zoom. Give the model one plain turn to answer from what it has
# (the same wrap-up the hand-written loop in the appendix uses). When the
# budget ran out, the runner has already appended the final assistant
# message and its tool results to the transcript, so we only append the
# wrap-up request itself (appending the final message again would
# duplicate its tool calls and the API would reject the transcript).
snapshot = {}
runner.set_messages_params(lambda params, _out=snapshot: (_out.update(params), params)[1])
transcript = list(snapshot["messages"])
if not (transcript and transcript[-1]["role"] == "user"):
transcript.append({"role": "assistant", "content": final.content})
transcript.append(
{
"role": "user",
"content": (
"You have used your zoom budget. Answer the original question now, "
"based on everything you have seen."
),
}
)
wrap_up = client.beta.messages.create(model=model, max_tokens=8192, messages=transcript)
final = wrap_up
answer = "\n".join(block.text for block in final.content if block.type == "text")
if verbose:
for block in final.content:
if block.type == "text":
print(f"[Assistant] {block.text}")
# Join every text block: long answers can arrive in several, and the
# answer is not always in the last one.
return answer演示：图表分析
让我们让 Claude 分析我们的图表。观察它如何使用放大工具来检查特定区域。
print(f"Question: {question}\n")
print("Claude's analysis:\n")
final_answer = ask_with_zoom_tool(chart_image, question)
print(f"\nGround truth (we drew it): {answer}")Question: What exact value is annotated at the peak of the Widget-C line?
Claude's analysis:[Tool] zoom({'x1': 260, 'y1': 35, 'x2': 450, 'y2': 80})<IPython.core.display.Image object>[Assistant] The annotation at the peak of the Widget-C line reads **"peak: 6,053 units"** — so the exact annotated value is **6,053 units**.
Ground truth (we drew it): peak: 6,053 units当全视图撒谎时
读取小文字只是失败模式的一半——另一半是几何关系。在同一图表的第 48 个月，Widget-A 和 Widget-E 线相距约 26 个单位：在此渲染中约为 8 像素，不到一个 28×28 块的一小部分，附近还有一个真正的交叉点使问题更加混乱。在这种间距下的排序问题在块粒度上物理上无法解决。
没有工具提问时，Claude 自信地选择了错误的线——在 40 次记录运行中有 36 次错误。有了放大工具，它会放大交叉区域（反复放大，每次裁剪都比上一次更紧凑）并得出正确答案——在 40 次中 40 次正确。
注意： 这个近乎平局的对比是针对
claude-fable-5（上面基准测试中最强的图表阅读器）调优的。较小的模型即使有工具也更频繁地出错——在我们的运行中，claude-sonnet-5在有工具可用的情况下 10 次尝试中只有 1 次正确判断了这个交叉点。放大恢复了像素；判断几乎接触的交叉点仍然需要模型最强的视觉推理能力。
question_2 = "At month 48, which product line has the higher value: Widget-A or Widget-E?"
# Without the tool: one plain Messages call, same image
view = prepare_image(chart_image)
response = client.messages.create(
model=MODEL,
max_tokens=8192,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": f"Answer the following question about the image below.\n\nThe question is: {question_2}",
},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": pil_to_base64(view),
},
},
],
}
],
)
print("WITHOUT the tool:")
print("".join(block.text for block in response.content if block.type == "text"))
print("\nWITH the zoom tool:")
ask_with_zoom_tool(chart_image, question_2)
higher = "Widget-E" if series["Widget-E"][48] > series["Widget-A"][48] else "Widget-A"
gap = abs(series["Widget-E"][48] - series["Widget-A"][48])
print(f"\nGround truth (from the data): {higher} is higher at month 48, by about {gap:.0f} units.")WITHOUT the tool:
**Widget-A** has the higher value at month 48. The blue Widget-A line is rising steeply at that point (around ~5,350 units), having just crossed above the purple Widget-E line, which is declining (around ~5,300 units). The two lines cross at roughly month 47, so by month 48 Widget-A is slightly above Widget-E.
WITH the zoom tool:[Tool] zoom({'x1': 980, 'y1': 230, 'x2': 1120, 'y2': 360})<IPython.core.display.Image object>[Tool] zoom({'x1': 1025, 'y1': 270, 'x2': 1075, 'y2': 320})<IPython.core.display.Image object>[Assistant] **Widget-E** has the higher value at month 48.
Zooming in on that region shows the purple Widget-E line (around ~5,310 units) sits just above the blue Widget-A line (around ~5,290 units) at month 48. The two lines cross shortly after — at roughly month 48–49 — after which Widget-A overtakes Widget-E, but at exactly month 48, Widget-E is still slightly higher.
Ground truth (from the data): Widget-E is higher at month 48, by about 26 units.以下是这个交互的实际过程——全视图、模型的放大调用，以及两条线最终分开的放大裁剪图：
多图像对话
你可以一次传递多张图像——例如，要求 Claude 比较两个图表。为每张图像提供索引、尺寸和坐标系，以便 Claude 可以指定它要放大哪张图像以及在哪里放大。
ask_with_zoom_tool([chart_image, other_chart], "Which of the two charts has more categories?")每个图像在提示中都标注了其索引和像素尺寸，因此 Claude 可以命名它想要的图像并使用正确的坐标空间。
总结
放大工具模式：
- 使用
resized_size()预调整你发送的图像大小，使 Claude 的像素坐标与你持有的图像 1:1 对应——并在提示中说明尺寸和左上角原点。 - 定义一个放大工具，接受绝对像素坐标。
- 从全分辨率原始图像裁剪，而不是从缩小后的副本，并将裁剪图放大到模型的图像预算，使小元素获得足够的视觉 token 来被读取。
- 让 Claude 决定何时何地放大。
这之所以有效，是因为 Claude 可以先看到完整图像，识别需要更仔细检查的区域，然后放大重新读取。权衡是：每次放大调用都会增加一次模型往返，放大后的图像消耗视觉 token，因此可以预期在细节相关的问题上获得更好的准确率，代价是更高的延迟和 token 使用量。（如果你在带代码执行的 Agent 框架中运行 Claude，你可能不需要构建这些：能够运行代码的 Agent 通常会按需编写自己的裁剪和放大脚本——同样的模式，自助实现。）
附录：手动实现循环
tool_runner 是一个便捷工具；它所做的一切都可以用普通的 client.messages.create 调用来实现，看一次其机制是值得的——这也是你自己控制每个边缘情况的地方：在工具调用出现时执行它们（即使在 token 上限的回合上），恢复服务器暂停的回合，当 token 上限的回合既没有产生工具调用也没有产生任何文本时给模型一个普通回合来回答，以及显式分派工具名称（未知的名称会返回错误结果，这样产生幻觉工具的模型可以恢复）。
下面的版本在常见路径上与运行器版本等效，有两个刻意的区别：运行器将迭代次数限制为 20 次、暂停回合重入限制为 5 次，而此版本会运行到模型停止——在生产环境中请添加你自己的上限。
from anthropic.types import (
ContentBlockParam,
ImageBlockParam,
MessageParam,
TextBlockParam,
ToolResultBlockParam,
)
def ask_with_zoom_tool_manual(
images, question: str, verbose: bool = True, model: str = MODEL
) -> str:
"""Ask Claude a question about one or more images, with the zoom tool available.
Accepts a single PIL image or a list of them. Returns Claude's final text.
"""
originals = images if isinstance(images, list) else [images]
views = [prepare_image(image) for image in originals]
content: list[ContentBlockParam] = [
{
"type": "text",
"text": f"Answer the following question about the image below.\n\nThe question is: {question}",
}
]
for i, view in enumerate(views):
label = f"Image {i}" if len(views) > 1 else "The image"
content.append({"type": "text", "text": f"\n{label} is {view.width}x{view.height} pixels:"})
content.append(
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": pil_to_base64(view),
},
}
)
content.append(
{
"type": "text",
"text": (
"\nPixel coordinates start at (0, 0) in the top-left corner, with x increasing "
"to the right and y increasing downward. Use the zoom tool to examine small "
"details like legends, axis labels, and closely spaced values."
),
}
)
messages: list[MessageParam] = [{"role": "user", "content": content}]
nudged = False
while True:
response = client.messages.create(
model=model, max_tokens=8192, tools=[ZOOM_TOOL], messages=messages
)
if verbose:
for block in response.content:
if block.type == "text":
print(f"[Assistant] {block.text}")
elif block.type == "tool_use":
print(f"[Tool] zoom({block.input})")
if response.stop_reason == "pause_turn":
# The model paused a long turn; hand it back as-is and let it continue.
messages.append({"role": "assistant", "content": response.content})
continue
tool_calls = [block for block in response.content if block.type == "tool_use"]
if tool_calls:
# Execute the tool calls and continue — keyed on the presence of the
# calls rather than on stop_reason, so a turn that emitted a complete
# call and then hit the token cap still gets its tool result.
messages.append({"role": "assistant", "content": response.content})
tool_results: list[ToolResultBlockParam] = []
for block in tool_calls:
if block.name != "zoom":
# Dispatch by name: when you add more tools, route each name
# to its handler here. Unknown names get an error result.
tool_results.append(
{
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Unknown tool: {block.name}",
"is_error": True,
}
)
continue
# block.input is typed loosely; the tool call's arguments arrive as a dict.
inputs = dict(cast(Mapping[str, Any], block.input))
image_index = inputs.pop("image_index", 0)
if not 0 <= image_index < len(originals):
result: list[TextBlockParam | ImageBlockParam] = [
{
"type": "text",
"text": f"Error: image_index {image_index} is out of range",
}
]
else:
try:
result = handle_zoom(
originals[image_index], views[image_index].size, **inputs
)
except TypeError as e:
# A malformed call (e.g. a missing coordinate, possible when a
# turn is cut off mid-call) gets an error result back, so the
# model can correct itself instead of crashing the loop.
result = [{"type": "text", "text": f"Error: invalid zoom arguments ({e})"}]
if verbose:
show_tool_result(result, show_text=False)
tool_results.append(
{"type": "tool_result", "tool_use_id": block.id, "content": result}
)
messages.append({"role": "user", "content": tool_results})
continue
if (
response.stop_reason == "max_tokens"
and not any(block.type == "text" for block in response.content)
and response.content
and not nudged
):
# The turn hit the token cap mid-thought: no answer text, no tool call.
# Hand the partial turn back and ask for a concise wrap-up (once).
nudged = True
messages.append({"role": "assistant", "content": response.content})
messages.append(
{
"role": "user",
"content": "You ran out of output tokens. State your final answer concisely now.",
}
)
continue
# Join every text block of the final turn: models that think can emit
# several, and the answer is not always in the last one.
return "\n".join(block.text for block in response.content if block.type == "text")
Tip
中文读者提示：本章节的代码和输出为英文原文。如需理解具体实现细节，可参考上方中文导读和代码注释。如有疑问，欢迎在 GitHub Issues 讨论。