Agentic Search 基准测试

复现 Claude 的 Agentic Search 评测基准

🌐 查看英文原文 | 源码 Notebook

Reproducing Claude’s Agentic Search Benchmark Scores

Claude 的已发布 agentic-search 评分 (DeepSearchQA, BrowseComp) 可以在公共 Messages API 上复现。关键在于 harness 配置：对于简短对话，一些 API 参数并不重要，但一旦代理运行了 30+ 工具调用，跨越数十万个 token，这些参数就变得至关重要。

第三方评估结果通常较低的一个常见原因是 harness 配置。本食谱将逐一介绍每个参数，并解释为什么它很重要。

成本 & 运行时间：本笔记本中的实时单元格运行 3 个演示问题（大约 5-10 分钟，API 花费几美元）。复现完整的 900 个问题的基准测试需要花费几百美元，并在中等并发下花费几个小时；本笔记本不会重新执行完整的运行。

本食谱结束时，您将能够：

构建使用程序化工具调用、服务器端压缩和任务预算的 agentic search 循环，以复现 Claude 的已发布 agentic-search 评分。
理解为什么每个配置选择对于长期 horizon 的 agentic 任务很重要。
适配相同的 harness 到 BrowseComp，或您自己的深度研究基准，通过交换数据集、几行配置和评分者。

先决条件

所需知识 - 熟悉 Messages API 和基本工具使用 - 了解 agentic loop 的概念（调用模型 → 处理工具使用 → 再次调用）

所需工具 - Python 3.11+ - anthropic >= 0.111.0 -Anthropic API密钥，可访问web_search、web_fetch和code_execution服务器工具（如何启用)

推荐 - 程序化工具调用食谱，其中介绍了我们在这里使用的模式 - 自动上下文压缩食谱，我们对此依赖很大

设置`python %%capture %pip install -U "anthropic>=0.111.0" pandas python-dotenv`

import time

import anthropic
import pandas as pd
from dotenv import load_dotenv
from utils.agentic_search import (
    SAMPLE_QUESTIONS,
    USER_PROMPT_TEMPLATE,
    extract_result_tag,
    grade_response,
    summarize_response,
)

load_dotenv()

MODEL = "claude-sonnet-5"
# Hold the grader model fixed so scores are comparable across the model under test.
GRADER_MODEL = "claude-opus-4-6"

# With server-side tools, a single streamed call can run for many minutes of
# in-sandbox tool execution before the first token arrives. We raise the read
# timeout to 60 minutes; with streaming this is a per-event ceiling, not a
# whole-response one, so it only matters for the gap before the first event.
client = anthropic.Anthropic(
    max_retries=20,
    timeout=anthropic.Timeout(5.0, read=3600.0, write=600.0, pool=600.0),
)
```## 任务

以下是三个DeepSearchQA格式的示例问题。每个问题都可以回答，但需要在搜索、阅读和交叉引用的几轮之后：

Here are three demo questions in the DeepSearchQA format. Each is answerable, but only after several rounds of searching, reading, and cross-referencing:

- [ ] Question 1
- [ ] Question 2
- [ ] Question 3

## 问题 1

Here is the first question:

::: callout-notice
**Question 1:**
What is the difference between RAG and traditional information retrieval systems?
:::

## 问题 2

Here is the second question:

::: callout-notice
**Question 2:**
How do embeddings help in natural language processing tasks?
:::

## 问题 3

Here is the third question:

::: callout-notice
**Question 3:**
What is the role of a token in a prompt for an Agent?
:::```python
for q in SAMPLE_QUESTIONS:
    print(f"[{q['example_id']}] ({q['answer_type']})")
    print(f"  Q: {q['problem']}")
    print(f"  A: {q['answer']}\n")

[demo-0] (Single Answer)
  Q: Among the chief executives of the four largest US banks by total assets as of Q1 2024, which one had held their CEO position the longest?
  A: Jamie Dimon

[demo-1] (Set Answer)
  Q: Which three countries had the largest installed offshore wind capacity at the end of 2023?
  A: China, United Kingdom, Germany

[demo-2] (Set Answer)
  Q: List the films that won the Palme d'Or at the Cannes Film Festival in 2021, 2022, and 2023, and name the director of each.
  A: Titane (Julia Ducournau), Triangle of Sadness (Ruben Östlund), Anatomy of a Fall (Justine Triet)
```我们将每个问题包裹在一个简短的提示中，要求Claude自由地规划并使用`<result>`标签来放置其最终答案，以便评分者可以干净地提取：

```markdown
We wrap each question in a short prompt that asks Claude to plan freely and put its final answer in `<result>` tags so the grader can extract it cleanly:
``````python
print(USER_PROMPT_TEMPLATE.format(question=SAMPLE_QUESTIONS[0]["problem"]))

I want you to answer the following question.

<question>Among the chief executives of the four largest US banks by total assets as of Q1 2024, which one had held their CEO position the longest?</question>

First plan out your response. This part can be as long as needed. You may need to run many searches, this is totally fine.

Then provide a short and concise answer in <result> tags. For questions expecting multiple answers, separate them with commas.
```模型需要规划、搜索网络、抓取和阅读页面、运行计算，并且一直这样做，直到它确信为止，然后输出一个干净的最终答案供我们评分。这句话的几乎每个部分都是朴素绑定默默失分的地方。我们将逐步构建这个循环。

> 上述三个问题是为这个食谱编写的（并非来自DeepSearchQA-900），以保持实时细胞快速且成本低廉。对于真实基准，请参阅下方的 *数据集*。

### 数据集

| 基准 | 大小 | 从哪里获取 |
|---|---|---|
| **BrowseComp** | 1,266 个问题 | [Wei 等人，2025](https://arxiv.org/html/2504.12516v1) (OpenAI) |
| **DeepSearchQA** | 900 个问题 | [Kaggle 基准](https://www.kaggle.com/benchmarks/google/dsqa) ([论文](https://arxiv.org/abs/2601.20975)) |

绑定期望每一行都有 `problem`、`answer` 和 `answer_type` 字段；在加载时相应地映射源模式。

## 1. 工具：程序性工具调用

最大的效率杠杆是 **[程序性工具调用](https://platform.claude.com/docs/en/agents-and-tools/tool-use/programmatic-tool-calling)** (PTC)：而不是Claude直接调用 `web_search` 和 `web_fetch` 并通过API往返每个结果，Claude在 `code_execution` 沙盒中编写Python，并从该代码内部调用搜索/抓取。只有代码打印的摘要返回到对话中。

对于深入研究，这是必不可少的。一个问题可以发出50+次抓取。没有PTC，每个页面的正文都会落在上下文中；有了PTC，模型在沙盒中读取它们，并只呈现它需要的部分。

我们通过将 `code_execution` 列为唯一可直接调用的工具，并将 `web_search`/`web_fetch` 标记为可从代码执行中调用（其结果排除在响应之外）来实现它：```python
TOOLS = [
    {
        "type": "code_execution_20260521",
        "name": "code_execution",
    },
    {
        "type": "web_search_20260318",
        "name": "web_search",
        "max_uses": 10_000,
        "allowed_callers": ["code_execution_20260521"],
        "response_inclusion": "excluded",
    },
    {
        "type": "web_fetch_20260318",
        "name": "web_fetch",
        "max_uses": 10_000,
        "max_content_tokens": 1_000_000,
        "allowed_callers": ["code_execution_20260521"],
        "response_inclusion": "excluded",
    },
]

BETAS = [
    "compact-2026-01-12",
    "task-budgets-2026-03-13",
]
```两个值得注意的细节：

- **`code_execution_20260521`** 使用了一个工具提示，揭示了沙盒的每个单元格的实时时钟限制，因此Claude将长时间运行的搜索拆分成更短的单元格，而不是编写一个超时的单个大循环。
- **`response_inclusion: "excluded"`** 将 `web_search` 和 `web_fetch` 结果保留在沙盒内；响应中的其他一切（`thinking`、`text`、`server_tool_use`、代码执行结果、`compaction`）都与 §3 中的 `model_dump()` 进行往返。

## 2. 思考、努力、预算和压缩

四个请求级别的参数告诉Claude *如何努力工作* 以及 *如何管理自己的上下文*：```python
THINKING = {"type": "adaptive"}

OUTPUT_CONFIG = {
    "effort": "max",
    "task_budget": {"type": "tokens", "total": 3_000_000},
}

# Match the system-card configuration. To reproduce a different model's score,
# use the effort tier and compaction trigger from that model's model card.
COMPACTION_TRIGGER = 200_000

COMPACT_INSTRUCTIONS = (
    "Your summary MUST begin by restating the user's ORIGINAL QUESTION "
    "verbatim and in full, wrapped in <original_question> and "
    "</original_question> tags. Then summarize the research progress so "
    "far: key sources found, data extracted, partial answers, and what "
    "remains to be done. Do NOT call any tools. Your summary MUST also "
    "include this instruction verbatim: 'Provide your final answer "
    "wrapped in <result> and </result> tags.'"
)

CONTEXT_MANAGEMENT = {
    "edits": [
        {
            "type": "compact_20260112",
            "trigger": {"type": "input_tokens", "value": COMPACTION_TRIGGER},
            "instructions": COMPACT_INSTRUCTIONS,
        }
    ]
}
```### 为什么 `COMPACT_INSTRUCTIONS` 如此重要

当对话达到触发阈值时，API会对其进行压缩：一个单独的模型调用总结历史，然后代理从总结处继续。默认的总结提示由设计上不针对任务；它不知道对于基准测试，原始问题和答案格式指令是绝对必须保留的两个东西。[自定义 `instructions`](https://platform.claude.com/docs/en/build-with-claude/compaction#custom-summarization-instructions) 是你告诉它的方式。

没有它们，压缩后的代理有一个总结 *它找到的内容* 但没有 *它被询问的内容*，在长问题中，它经常要求用户重述问题，这得零分。

效果随着压缩触发的频率而扩大：在低触发（一个单独的问题可能被压缩多次）时最大，在这里使用的 200K 触发时较小，但无论如何都是最便宜的保险。

> **注意**：最新的 Claude 模型卡片报告了没有压缩的 DeepSearchQA（该模型有一个 1M-token 的上下文窗口，因此一个 1M 预算的任务可以放入一个窗口），而 BrowseComp 报告的是带有压缩的，总预算为 3M。我们在这里使用 3M 预算的 BrowseComp 配置，以便压缩有意义；在 DeepSearchQA 中，大多数问题仍然在 1M 以下完成，并且从未触发它。

其他三个：

| 参数 | 为什么 |
|---|---|
| `thinking: adaptive` | 允许 Claude 在结果令人惊讶时在工具调用之间进行思考，在结果不令人惊讶时跳过思考 |
| `effort` | 模型卡片上的数字是在模型最高努力级别上报告的 |
| `task_budget.total` | 告诉 Claude 在所有回合和压缩中的累积输出预算，因此它可以调整自己的步伐，而不是过早放弃 |


## 3. 往返循环
使用服务器端工具，每个API调用可以在内部运行多个工具迭代，然后返回 `stop_reason="pause_turn"` 以进行检查点。我们在客户端的工作只是：获取响应内容，将其追加为助手回合，并再次调用。我们持续进行，直到 `stop_reason="end_turn"`。

一个辅助工具在回合间保持请求体整洁：

::: callout-attention
请注意，以下代码示例仅供参考，实际应用时请根据具体情况进行调整。
:::

```python
def process_api_call(response):
    # 处理API调用响应
    pass

def append_assistant_turn(response):
    # 追加助手回合
    pass

# 主循环
while True:
    response = make_api_call()
    process_api_call(response)
    append_assistant_turn(response)
    if response['stop_reason'] == 'end_turn':
        break
``````python
def truncate_to_last_compaction(messages: list[dict]) -> list[dict]:
    """Drop history that precedes the most recent compaction block.

    The server already discarded that history when it compacted; resending it is
    pure overhead and on long runs can exceed the 32 MB request-body limit.
    `messages[0]` (the original user prompt) is always kept so the request still
    starts with a user turn.
    """
    for mi in range(len(messages) - 1, 0, -1):
        content = messages[mi].get("content")
        if not isinstance(content, list):
            continue
        for bi in range(len(content) - 1, -1, -1):
            blk = content[bi]
            if isinstance(blk, dict) and blk.get("type") == "compaction":
                trimmed = {**messages[mi], "content": content[bi:]}
                return [messages[0], trimmed, *messages[mi + 1 :]]
    return messages
```现在来看这个循环本身。它是一个单独的函数，虽然很长，但每行都在做真实基准测试运行所需的事情：


```python
def benchmark_run():
    # ... 函数内容 ...
``````python
def sample(question: str, *, max_turns: int = 100) -> dict:
    """Run one question to completion through the agentic search loop."""
    # cache_control on the user message turns on prompt caching for the whole
    # request, so the server-side tool loop caches between iterations.
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": USER_PROMPT_TEMPLATE.format(question=question),
                    "cache_control": {"type": "ephemeral"},
                }
            ],
        }
    ]
    request = {
        "model": MODEL,
        "max_tokens": 64_000,
        "tools": TOOLS,
        "betas": BETAS,
        "thinking": THINKING,
        "output_config": OUTPUT_CONFIG,
        "context_management": CONTEXT_MANAGEMENT,
        "messages": messages,
    }

    usage = {"input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0}
    transcript: list[str] = []
    tool_calls = 0
    start = time.time()

    for turn in range(max_turns):
        # Stream so the per-event read-timeout applies, not a per-response one.
        with client.beta.messages.stream(**request) as stream:
            response = stream.get_final_message()

        usage["input_tokens"] += response.usage.input_tokens
        usage["output_tokens"] += response.usage.output_tokens
        usage["cache_read_input_tokens"] += (
            getattr(response.usage, "cache_read_input_tokens", 0) or 0
        )
        tool_calls += sum(
            1 for b in response.content if getattr(b, "type", "") == "server_tool_use"
        )
        transcript.append(f"--- turn {turn} ---\n" + summarize_response(response))

        # Persist the code-execution container so state survives across turns.
        if getattr(response, "container", None) is not None:
            request["container"] = response.container.id

        # Round-trip: append assistant content, drop pre-compaction history.
        messages.append(
            {
                "role": "assistant",
                "content": [b.model_dump(exclude_none=True) for b in response.content],
            }
        )
        messages[:] = truncate_to_last_compaction(messages)

        if response.stop_reason == "end_turn":
            final_text = "".join(
                b.text for b in response.content if getattr(b, "type", "") == "text"
            )
            return {
                "text": final_text,
                "result": extract_result_tag(final_text),
                "turns": turn + 1,
                "tool_calls": tool_calls,
                "elapsed_s": time.time() - start,
                "usage": usage,
                "transcript": transcript,
            }
        if response.stop_reason not in ("pause_turn", "max_tokens"):
            raise RuntimeError(f"unexpected stop_reason={response.stop_reason}")

    raise RuntimeError(f"did not finish within {max_turns} turns")
```### 重试

SDK 在连接错误、`408`、`409`、`429` 和 `≥500` 响应时自动重试，并使用指数退避策略，同时尊重API在速率限制响应中发送的 `Retry-After` 头部信息。默认重试次数为2次；为了进行完整的基准测试运行，我们在客户端（如上方的 *Setup* 中）将 `max_retries=20` 设置为最大重试次数，这样运行过程中的短暂过载不会导致一个问题得分为零。

> **检测无声的工具节流**：服务器工具的速率限制与模型令牌限制是分开的。耗尽它们不会引发API错误；`too_many_requests` 会显示为响应中的工具结果错误，Claude通常会在沙箱内重试。如果完整的基准测试运行得分意外地低且没有异常，请检查 `result["transcript"]` 以查找打印 `rate_limit_error` 或 `too_many_requests` 的代码执行单元格。客户端的解决方案是降低并发量（或联系Anthropic支持以获得更高的工具速率限制）；客户端的重试不会有所帮助，因为API调用已返回200。
>
> 同样，将每个问题的 `sample()` 调用包裹在 `try/except` 中，这样单个 `stop_reason="refusal"` 或不可恢复的错误会将该问题得分为零，而不是终止整个运行。

## 4. 完成一个问题

让我们运行第一个问题。请期待这个单元格需要一分钟左右的时间：在单个 `stream()` 调用背后，Claude正在多次编写和执行搜索代码。```python
result = sample(SAMPLE_QUESTIONS[0]["problem"])

print(f"tool_calls={result['tool_calls']}  elapsed={result['elapsed_s']:.0f}s")
print(
    f"tokens: in={result['usage']['input_tokens']:,}  out={result['usage']['output_tokens']:,}  "
    f"cache_read={result['usage']['cache_read_input_tokens']:,}"
)
print()
print("Final <result>:", result["result"])

tool_calls=4  elapsed=56s
tokens: in=130  out=3,842  cache_read=39,703

Final <result>: **Jamie Dimon**, CEO of **JPMorgan Chase**, had held his CEO position the longest among the chief executives of the four largest US banks by total assets (JPMorgan Chase, Bank of America, Citigroup, and Wells Fargo) as of Q1 2024. Dimon became JPMorgan Chase's CEO on January 1, 2006 — roughly 18 years by Q1 2024 — compared to Brian Moynihan (Bank of America CEO since January 2010), Charles Scharf (Wells Fargo CEO since October 2019), and Jane Fraser (Citigroup CEO since March 2021).
```原文：

The transcript shows the shape of the work. Each turn is a `pause_turn` checkpoint after a batch of in-sandbox tool calls:

翻译：

转录显示了工作的形状。每个回合都是在一系列沙盒工具调用之后的 `pause_turn` 检查点：```python
for line in result["transcript"]:
    print(line + "\n")

--- turn 0 ---
  [thinking]  963 chars
  [text]      I'll research this step by step: first identifying the four largest US banks by total assets as of Q
  [tool_use]  code_execution
  [code_execution_tool_result]
  [thinking]  82 chars
  [tool_use]  code_execution
  [code_execution_tool_result]
  [thinking]  511 chars
  [tool_use]  code_execution
  [code_execution_tool_result]
  [thinking]  1365 chars
  [tool_use]  code_execution
  [code_execution_tool_result]
  [thinking]  700 chars
  [text]      Based on my research, I now have sufficient information to answer this question.  **Summary of findi
  [text]      "JPMorgan Chase & Co., the largest US bank by total assets, reported a sequential increase of $215.3
  [text]       were the other top banks, a ranking that held consistently across 2024.  Their respective CEOs and 
  [text]      Dimon became CEO on January 1, 2006 and one year later also became Chairman of the Board.
  [text]        2. **Bank of America – Brian Moynihan**: 
  [text]      Moynihan took office on January 1, 2010, and also is a member of the Bank of America board of direct
  [text]        3. **Wells Fargo – Charles Scharf**: 
  [text]      The current CEO and President is Charles W. Scharf, appointed in September 2019
  [text]      , with his effective start date being October 21, 2019 according to Wells Fargo's official announcem
  [text]      Jane Fraser is a British-American banking executive who has been chief executive officer (CEO) of Ci
  [text]      , becoming the first woman to lead a major U.S. bank.  Comparing the start dates — 2006 for Dimon, 2
  usage: in=130 out=3,842 cache_read=39,703 stop=end_turn
```## 5. 评分

基准评分使用以模型为评判者的F1评分器。评分器看到问题、黄金答案和提取的 `<result>` 文本，然后报告哪些黄金项被找到以及哪些额外的项出现在黄金集中。据此，我们计算精确度、召回率和F1。

有两件事需要做对：

- **评分 `<result>` 标签，而不是整个响应。** 完整的响应包括规划性文字和搜索摘要；将这些内容提供给评分器会导致误报和漏报增加。`extract_result_tag()` 只提取最终的答案。
- **保持评分模型固定。** 我们使用 `GRADER_MODEL = "claude-opus-4-6"`，无论 `MODEL` 如何。使用测试模型进行评分会使不同模型之间的分数不可比。

评分器的提示非常直接：

~~~text
您的任务是评估给定的响应是否得到了正确答案。

问题：<question>{question}</question>
正确答案（类型：{answer_type}）：<correct_answer>{answer}</correct_answer>
要评估的响应：<response>{response}</response>

对于每个预期的答案项，指出它是否出现在响应中。
然后列出响应中不在正确答案列表中的任何答案。
措辞不需要完全一致。

以以下精确的XML格式回复：
<evaluation>
  <explanation>一句话</explanation>
  <correctness_details>
    <item answer="expected_item_1" correct="true|false"/>
  </correctness_details>
  <excessive_answers>
    <item>extra_item_if_any</item>
  </excessive_answers>
</evaluation>
~~~

解析和精确度/召回率/F1的算术运算位于 `utils/agentic_search.py` 中；这里我们只是调用它：```python
grade = grade_response(client, GRADER_MODEL, SAMPLE_QUESTIONS[0], result["text"])
print(f"answer    : {grade['extracted_answer']}")
print(f"precision : {grade['precision']:.2f}")
print(f"recall    : {grade['recall']:.2f}")
print(f"F1        : {grade['f1']:.2f}")

answer    : **Jamie Dimon**, CEO of **JPMorgan Chase**, had held his CEO position the longest among the chief executives of the four largest US banks by total assets (JPMorgan Chase, Bank of America, Citigroup, and Wells Fargo) as of Q1 2024. Dimon became JPMorgan Chase's CEO on January 1, 2006 — roughly 18 years by Q1 2024 — compared to Brian Moynihan (Bank of America CEO since January 2010), Charles Scharf (Wells Fargo CEO since October 2019), and Jane Fraser (Citigroup CEO since March 2021).
precision : 1.00
recall    : 1.00
F1        : 1.00
```## 6. 运行演示集

现在所有三个问题，使用情况和F1分数的比较表格。（要运行完整的900个问题的基准测试，将`SAMPLE_QUESTIONS`替换为完整的数据集并添加并发；见*下一步*。）


## 7. 使用指南

### 7.1 安装依赖

首先，确保你已经安装了以下依赖项：

- Python 3.8+
- `pip` (Python包管理器)

```bash
pip install -r requirements.txt

7.2 运行示例

以下是如何运行示例代码的步骤：

import my_module

# 初始化模型
model = my_module.Model()

# 加载数据
data = my_module.load_data('data.json')

# 训练模型
model.train(data)

# 预测
predictions = model.predict(data)

# 打印结果
print(predictions)

::: callout-notice 注意请确保你的数据格式与示例代码中的data.json相匹配。 :::

7.3 API参考

以下是对API的简要说明：

my_module.Model()：创建一个新的模型实例。
my_module.load_data(file_path)：从文件中加载数据。
model.train(data)：使用数据训练模型。
model.predict(data)：使用模型进行预测。

更多详细信息，请参阅官方文档。

::: callout-notice 注意 API可能会在未来的版本中发生变化，请确保查看最新的文档。 :::```python rows = [] for q in SAMPLE_QUESTIONS: r = sample(q[“problem”]) g = grade_response(client, GRADER_MODEL, q, r[“text”]) rows.append( { “id”: q[“example_id”], “tool_calls”: r[“tool_calls”], “in_tokens”: r[“usage”][“input_tokens”], “out_tokens”: r[“usage”][“output_tokens”], “f1”: g[“f1”], “answer”: g[“extracted_answer”][:60], } )

df = pd.DataFrame(rows) print(df.to_string(index=False)) print(f”F1: {df[‘f1’].mean():.2f}“) # in_tokens is the uncached billed input; with caching on, most of the actual # context is cache reads (tracked separately in usage.cache_read_input_tokens). print(f”Total tokens: {df[‘in_tokens’].sum():,} in / {df[‘out_tokens’].sum():,} out”)

id  tool_calls  in_tokens  out_tokens  f1                                                        answer

demo-0 11 312 10503 1.0 Jamie Dimon, CEO of JPMorgan Chase (the largest US b demo-1 7 208 9768 1.0 The three countries with the largest installed offshore wind demo-2 3 104 1592 1.0 - 2021: Titane — directed by Julia Ducournau- **2022*

Mean F1: 1.00 Total tokens: 624 in / 21,863 out ```## 7. 扩展到完整基准

在DeepSearchQA-900和BrowseComp-1266完整集上运行此笔记本的配置，可以重现已发布的模型卡片分数。有关跨模型的已发布代理搜索分数，请参阅每个模型的Claude模型卡片。

8. 适配BrowseComp

BrowseComp问题为单答案（“哪一部是1995年的电影……”）而非集合值，因此唯一的 harness 更改是评分器：

请注意，这里使用了callout语法来突出显示某些信息。 :::```python from utils.agentic_search import grade_browsecomp

Grading: a single-letter judge (A=correct, B=incorrect, C=abstain).

accuracy = fraction of “A” verdicts.

bc_grade = grade_browsecomp(client, GRADER_MODEL, SAMPLE_QUESTIONS[0], result[“text”]) print(bc_grade)

{‘example_id’: ‘demo-0’, ‘extracted_answer’: “Jamie Dimon, CEO of JPMorgan Chase, had held his CEO position the longest among the chief executives of the four largest US banks by total assets (JPMorgan Chase, Bank of America, Citigroup, and Wells Fargo) as of Q1 2024. Dimon became JPMorgan Chase’s CEO on January 1, 2006 — roughly 18 years by Q1 2024 — compared to Brian Moynihan (Bank of America CEO since January 2010), Charles Scharf (Wells Fargo CEO since October 2019), and Jane Fraser (Citigroup CEO since March 2021).”, ‘grader_letter’: ‘A’, ‘accuracy’: 1.0} ``工具、beta版本和sample()`循环保持不变。如果您想进行严格的复现，请查看模型卡片以获取确切的努力层级和压缩触发器 BrowseComp 的报告；§2 中的值是一个很好的默认设置。

这就是关键点：一旦循环正确，更换基准测试就是一个数据集、几行配置和评分者的问题了。

概述

您构建了一个具有以下特征的代理搜索循环：

从代码执行中程序化地调用web_search和web_fetch，因此50多页的抓取不会淹没上下文（§1）。
告诉 Claude 工作难度（effort，task_budget）和思考方式（adaptive）（§2）。
通过在压缩提示中重新表述问题来生存压缩，这是在高压缩频率下最大的杠杆（§2）。
正确往返pause_turn响应，持久化代码容器并修剪压缩前的历史记录（§3）。
使用以模型为评判者的 F1 评分器进行评分，并复现了发布的 DeepSearchQA 分数（§5–7）。

下一步

扩展到完整基准测试：将sample()包裹在concurrent.futures.ThreadPoolExecutor中（客户端是线程安全的）；SDK 内置的重试已经涵盖了429/529。在并发50的情况下，DeepSearchQA-900大约需要2小时完成。
尝试多代理变体：异步多代理编排食谱展示了N个代理团队如何以更低的延迟超过单个代理循环的准确性。
将其移植到您自己的基准测试：合同是[{problem, answer, answer_type}]行，几行配置和评分器函数。循环本身没有变化。
阅读模型卡片：每个模型的 Claude 模型卡片记录了每个发布的基准测试数字所使用的配置。

Tip

中文读者提示：本章节的代码和输出为英文原文。如需理解具体实现细节，可参考上方中文导读和代码注释。如有疑问，欢迎在 GitHub Issues 讨论。

Reproducing Claude’s Agentic Search Benchmark Scores

本食谱结束时，您将能够：

先决条件

设置python %%capture %pip install -U "anthropic>=0.111.0" pandas python-dotenv

7.2 运行示例

7.3 API参考

8. 适配BrowseComp

Grading: a single-letter judge (A=correct, B=incorrect, C=abstain).

accuracy = fraction of “A” verdicts.

概述

下一步

设置`python %%capture %pip install -U "anthropic>=0.111.0" pandas python-dotenv`