CMA：结果评分验证

使用评分器验证 Agent 输出质量

🌐 查看英文原文 | 源码 Notebook

结果：自我验证的 Agent

Agent 擅长产生看起来已完成的东西。向一个 Agent 要一份带引用的研究简报，你会得到一份带有脚注的整洁文档。仔细看，通常有改进空间：某个主题覆盖不足，某段引文偏离了原文，某条引用依赖新闻稿而不是原始文件。发现这些问题需要一个手动审查循环：你阅读输出，发现不对的地方，然后再次提示。而你在这些轮次中说的大部分内容是你在 Agent 开始之前就可以写下来的反馈。

Claude Managed Agents 中的 Outcomes 给 session 提供了第二个 Agent，其唯一工作是检查。你编写一个评分标准，详细说明”完成”是什么样的以及如何验证它，平台在自己的上下文窗口中配置一个评分器。评分器看不到写作者的推理，也不知道采取了什么捷径。在每轮写作者之后，它根据评分标准重新阅读产物，要么通过它，要么返回一个按标准的差距列表。写作者修订，循环再次运行，直到你设置的上限。

在本指南中，你将看到该循环端到端运行。写作者起草一份关于 EV 快充经济性的一页简报，带有来自其来源的逐字引用。评分器独立获取每个引用的 URL，在每个页面中搜索引用的字符串，检查引用是否确实支持该声明，并根据七项检查表对覆盖范围评分。你将看到它捕获一个真实问题（引用了需要 10-K 文件的新闻稿附件），你将看到写作者去寻找正确的文档。

你将学到什么

编写评分器可以执行的评分标准
使用 Outcome 启动 session 并让 Agent 朝其工作
跟踪评分-修订循环并阅读评分器的反馈
认识何时 Outcomes 是正确的工具

1. 设置环境

首先，让我们安装 SDK 并设置 Anthropic 客户端。define_outcome 事件和 outcome 评估 span 类型是 Managed Agents beta 的一部分。

%%capture
%pip install -q anthropic python-dotenv

import os
import re
import time

import anthropic
from dotenv import load_dotenv

load_dotenv()

BETAS = ["managed-agents-2026-04-01"]
MODEL = os.environ.get("COOKBOOK_MODEL", "claude-sonnet-4-6")
client = anthropic.Anthropic()

2. 创建写作者并启动 session

接下来，我们将创建写作者 agent 并打开一个 session。写作者的系统提示要求它引用不超过六个来源，每个都带有一段简短的逐字引用，以便评分器有具体的内容可以检查。

env = client.beta.environments.create(
    name="research-brief",
    config={"type": "anthropic_cloud", "networking": {"type": "unrestricted"}},
)

writer = client.beta.agents.create(
    name="Research Analyst",
    model=MODEL,
    system="""You are a research analyst. You write one-page business briefs.

Cite every factual claim with an inline footnote [n]. End the brief with a Sources section in this exact format, one entry per line:

[n] "verbatim quote from the page, 25 words or fewer" - Title - URL

Only cite pages you actually fetched and read. The quote must be copied character-for-character from the page. Cite no more than 6 sources total. Pick the strongest; do not pad. Save the brief to /mnt/session/outputs/brief.md.""",
    tools=[
        {
            "type": "agent_toolset_20260401",
            "configs": [
                {"name": "web_search"},
                {"name": "web_fetch"},
                {"name": "read"},
                {"name": "write"},
            ],
        }
    ],
    betas=BETAS,
)

session = client.beta.sessions.create(
    agent={"type": "agent", "id": writer.id, "version": writer.version},
    environment_id=env.id,
    title="Brief: EV fast-charging unit economics",
    betas=BETAS,
)
print(f"Session {session.id}")

Session sesn_011CakRYQjc4NMXdRy4qKZy5

3. 编写评分器可以执行的评分标准

define_outcome 事件给 session 两样东西：写作者阅读的交付物描述，以及评分器阅读的评分标准。每轮写作者之后，平台使用与写作者相同的模型和工具启动一个新的评分器，给它评分标准，让它检查产物。评分器返回每个标准的判定，如果有任何失败，解释直接返回给写作者进行下一次修订。

评分标准是你对评分器的唯一控制手段，其措辞决定了评分器是否真的检查任何东西。默认的失败模式是评分器批准一切。一个说“检查简报是否覆盖了需求电费”的评分标准让评分器浏览简报，看到一段关于需求电费的内容，然后写一句话确认它在那里。它可以在不打开任何来源的情况下完成所有这些。大多数初稿在此基础上通过，循环从不运行。一个说“打开简报，找到需求电费部分，确认它说明了 $/kW 数字或运营成本的百分比”的评分标准迫使评分器产生证据。那个评分器才能捕获冒充 10-K 的新闻稿。

每个评分标准中都值得做的几件事：

原则	实践中
使每个标准可检查	任务说”指定运营商的经济学”。评分标准将其具体化：来自 10-K 或 10-Q 的 GAAP 净亏损，引用 `sec.gov` 上的文件。评分标准应该始终比任务更具体。
让评分器赢得 `satisfied`	在评分器通过任何内容之前要求具体证据（获取的页面、追踪的公式、`file:line` 引用）。过于严格的评分器会让你多一轮循环。过于宽松的评分器在坏版本仍然存在时结束循环。
描述目标，而不是步骤	规定特定命令的评分标准在不可用时会静默失败，而你关心的检查从不运行。定义什么算作证明；评分器拥有写作者的完整工具集，可以找到它。
预判写作者的捷径	“不要通过镜像、转载或搜索片段来佐证。” 没有那行，一个死来源会被替换为一个抓取页面，评分器仍然通过它。
强制反馈格式	评分器的解释是写作者的唯一信号。要求一行评分板，然后每个失败一个要点，说明什么有问题以及该怎么做。
告诉评分器忽略什么	没有排除列表，评分器会在风格细节、已有问题和范围蔓延上纠结。明确什么是超出范围的，并让它在提出之前自我检查每个发现。

还没有评分标准？ 给 Claude 一个已知良好的产物示例，让它分析什么使它好，然后将该分析转化为标准。这种中间立场可靠地胜过从空白页编写标准。

这是我们简报的任务和评分标准。并排阅读它们，注意评分标准具体多少。它添加任务所没有的细节的每个地方都是评分器可以实际运行的检查。

TASK = """Write a brief on the unit economics of public DC fast charging in the United States.
The brief should cover:
  1. Capex range
  2. Demand charges
  3. Utilization breakeven
  4. Subsidy programs
  5. Named-operator economics
  6. A contrarian or skeptical source
  7. Hardware vs install cost split
"""


RUBRIC = """
You are reviewing a research brief at /mnt/session/outputs/brief.md against a coverage checklist and verifying its citations. The writer was told the seven topics to cover; this rubric defines what counts as sufficient coverage for each topic, and how to verify citations.

COVERAGE CHECKLIST. Each item has a specific area:
  1. Capex range: a dollar range for installed cost per DC fast-charging stall or station.
  2. Demand charges: quantified impact on opex (a $/kW figure or a % of operating cost).
  3. Utilization breakeven: a breakeven or target utilization threshold (% or kWh/day).
  4. Subsidy programs: NEVI or another public funding program, named.
  5. Named operator: the GAAP net income or net loss from a specific public charging operator's most recent 10-K or 10-Q, and the citation for it must be the SEC filing itself (sec.gov), not a press release, earnings-call recap, or news article.
  6. Contrarian source: at least one cited source whose thesis is that the economics are unfavorable or structurally challenged.
  7. Cost split: a hardware vs soft-cost (install, permitting, grid) breakdown or ratio.

CITATION CHECK. For every [n] entry in the Sources section:
  a. LIVE: Fetch the URL with web_fetch. Mark LIVE only if web_fetch returns the readable page directly. Mark DEAD if 404, parked, login-walled, paywalled, returns a bot-block/403, or renders only via JavaScript. Do NOT corroborate via mirrors, reposts, or search snippets; the cited URL itself must fetch.
  b. VERBATIM: Search the fetched page for the quoted string. Mark QUOTE_MATCH if the exact string appears (treat curly vs straight quotes as equivalent); NOT_FOUND otherwise.
  c. SUPPORTS CLAIM: Mark SUPPORTS_CLAIM if the quoted passage actually backs the claim it's cited on in the brief; UNSUPPORTED if it's tangential, contradicts the claim, or is just a general statement of fact.

OUTPUT FORMAT:

Line 1: Coverage N/7. Citations M/K verified.

Then, for each failed item in the coverage checklist, create a new bullet, name the item and say what specific bar it failed in one sentence max per bullet. For example: "Item 3 Utilization breakeven - MISSING. <what's missing>".

Then, for each failed citation, create a new bullet with the format: "[n] domain - REASON. <what's wrong and what to do>". One sentence max per bullet. For example: "[3] evgo.com - DEAD. The URL returns a 403 error and appears to be behind a bot block. No mirrors or reposts; the cited URL itself must fetch."
"""

client.beta.sessions.events.send(
    session.id,
    betas=BETAS,
    events=[
        {
            "type": "user.define_outcome",
            "description": TASK,
            "rubric": {"type": "text", "content": RUBRIC},
            "max_iterations": 5,
        },
    ],
)

BetaManagedAgentsSendSessionEvents(data=[BetaManagedAgentsUserDefineOutcomeEvent(id='sevt_011CakRYTKNkXJN7eh4kRs1r', description='Write a brief on the unit economics of public DC fast charging in the United States.\nThe brief should cover:\n  1. Capex range\n  2. Demand charges\n  3. Utilization breakeven\n  4. Subsidy programs\n  5. Named-operator economics\n  6. A contrarian or skeptical source\n  7. Hardware vs install cost split\n', max_iterations=5, outcome_id='outc_011CakRYTKNkJ83CBexcz13e', processed_at=datetime.datetime(2026, 5, 6, 1, 10, 30, 512206, tzinfo=TzInfo(0)), rubric=BetaManagedAgentsTextRubric(content='\nYou are reviewing a research brief at /mnt/session/outputs/brief.md against a coverage checklist and verifying its citations. The writer was told the seven topics to cover; this rubric defines what counts as sufficient coverage for each topic, and how to verify citations.\n\nCOVERAGE CHECKLIST. Each item has a specific area:\n  1. Capex range: a dollar range for installed cost per DC fast-charging stall or station.\n  2. Demand charges: quantified impact on opex (a $/kW figure or a % of operating cost).\n  3. Utilization breakeven: a breakeven or target utilization threshold (% or kWh/day).\n  4. Subsidy programs: NEVI or another public funding program, named.\n  5. Named operator: the GAAP net income or net loss from a specific public charging operator\'s most recent 10-K or 10-Q, and the citation for it must be the SEC filing itself (sec.gov), not a press release, earnings-call recap, or news article.\n  6. Contrarian source: at least one cited source whose thesis is that the economics are unfavorable or structurally challenged.\n  7. Cost split: a hardware vs soft-cost (install, permitting, grid) breakdown or ratio.\n\nCITATION CHECK. For every [n] entry in the Sources section:\n  a. LIVE: Fetch the URL with web_fetch. Mark LIVE only if web_fetch returns the readable page directly. Mark DEAD if 404, parked, login-walled, paywalled, returns a bot-block/403, or renders only via JavaScript. Do NOT corroborate via mirrors, reposts, or search snippets; the cited URL itself must fetch.\n  b. VERBATIM: Search the fetched page for the quoted string. Mark QUOTE_MATCH if the exact string appears (treat curly vs straight quotes as equivalent); NOT_FOUND otherwise.\n  c. SUPPORTS CLAIM: Mark SUPPORTS_CLAIM if the quoted passage actually backs the claim it\'s cited on in the brief; UNSUPPORTED if it\'s tangential, contradicts the claim, or is just a general statement of fact.\n\nOUTPUT FORMAT: \n\nLine 1: Coverage N/7. Citations M/K verified.\n\nThen, for each failed item in the coverage checklist, create a new bullet, name the item and say what specific bar it failed in one sentence max per bullet. For example: "Item 3 Utilization breakeven - MISSING. <what\'s missing>".\n\nThen, for each failed citation, create a new bullet with the format: "[n] domain - REASON. <what\'s wrong and what to do>". One sentence max per bullet. For example: "[3] evgo.com - DEAD. The URL returns a 403 error and appears to be behind a bot block. No mirrors or reposts; the cited URL itself must fetch."\n', type='text'), type='user.define_outcome')])

关于上面调用的几点说明：

描述和评分标准有不同的工作。 描述告诉写作者做什么。评分标准告诉评分器如何检查它。它们应该在产物的位置和格式上达成一致；如果它们相互矛盾（例如描述要求内联输出而评分标准评分 /mnt/session/outputs/ 中的文件），循环返回 failed 而不是纠结。
max_iterations 默认为 3（最大 20）。 我们设置为 5，因为该评分标准很严格，写作者需要空间。如果每次运行都达到上限，评分器每次发现相同类型的问题，写作者无法根据反馈采取行动，你在为不收敛的迭代付费。
内联评分标准 vs. 文件评分标准。 对于 notebook，内联文本可以。在生产环境中，你会通过 Files API 上传评分标准一次并传递 rubric: {"type": "file", "file_id": ...}，以便它可跨 session 复用并像代码一样可审查。

为什么不直接把评分标准放在系统提示中？ 你可以，它有助于写作者更好地瞄准目标。但知道标准的写作者仍然在给自己的工作打分。只要它认为自己通过了就会说通过了，它不会回去重新获取已经引用的 URL 或注意到它记忆中的引文与页面上的引文略有不同。评分器别无选择，只能做这些检查。它以全新的上下文窗口打开，只有评分标准和产物，平台不允许循环继续，直到它对每个标准产生判定。无论你写得多么好，你都无法从单个提示中获得这种分离。

4. 观看审查循环

让我们流式传输事件并在发生时渲染每个阶段。我们将在写作者完成草稿时打印横幅，并在每次评估后显示评分器的反馈。

下面循环的两个显示辅助函数：banner 绘制一个带标签的分隔符，render_feedback 去除服务器包裹每个评分器解释的样板文本。

HR = "━" * 46


def banner(label, tag=""):
    print(f"\n{HR}\n{label}  {tag}".rstrip())


def render_feedback(fb: str):
    # Strip the server's per-criterion wrapper and trailer.
    s = re.sub(
        r"^An independent grader found.*?:\n\n- .*?\((?:partially |not )?met\): ",
        "",
        fb,
        count=1,
        flags=re.S,
    )
    s = re.sub(r"\n\nPlease revise your work.*$", "", s, flags=re.S)
    print(s)

TERMINAL = {"satisfied", "max_iterations_reached", "failed", "interrupted"}
t0, res, iters = time.time(), None, 0
n_search, last_len = 0, 0

with client.beta.sessions.events.stream(session.id, betas=BETAS) as stream:
    for ev in stream:
        if ev.type == "agent.tool_use":
            if ev.name in ("web_search", "web_fetch"):
                n_search += 1
            if ev.name == "write" and ev.input["file_path"].endswith("brief.md"):
                last_len = len(ev.input["content"])
        elif ev.type == "span.outcome_evaluation_start":
            banner("writer · " + ("draft" if iters == 0 else f"revision {iters}"))
            print(f"searched/fetched {n_search}× · wrote brief.md ({last_len:,} chars)")
            n_search = 0
        elif ev.type == "span.outcome_evaluation_end":
            res = ev.result
            banner(
                f"grader · pass {iters}",
                "✓ satisfied" if res == "satisfied" else "⟳ needs_revision",
            )
            render_feedback(ev.explanation)
            iters += 1
            if res in TERMINAL:
                break

m, s = divmod(int(time.time() - t0), 60)
print(f"\ndone: {res} after {iters} pass{'es' if iters != 1 else ''} · {m}m {s:02d}s")


━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
writer · draft
searched/fetched 18× · wrote brief.md (7,607 chars)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
grader · pass 0  ⟳ needs_revision
The brief covers 5 of 7 required topics adequately. Item 2 (Demand charges) fails the quantification bar: the brief describes demand charges qualitatively as the 'single largest operational wildcard' but never states a $/kW figure (McKinsey's footnote of $20/kW is never quoted in the text) or a % of operating cost. Item 5 (Named operator) fails the citation requirement: EVgo's Q1 2024 net loss of $28.2M is cited to evchargingstations.com [6], a third-party news article, not an SEC filing from sec.gov as the rubric explicitly requires. All 6 citations are LIVE, and all 6 quoted strings match the fetched pages and support the claims they are attached to — but [6] is the wrong source type for the Named Operator criterion.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
writer · revision 1
searched/fetched 6× · wrote brief.md (7,989 chars)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
grader · pass 1  ⟳ needs_revision
Six of seven coverage items are fully met: (1) capex range with hardware $45K–$100K and install $40K–$150K per charger; (2) demand charges quantified at $20/kW and $9,600/month for a 480kW station; (3) utilization breakeven at 20% vs. 7.5% national average; (4) NEVI Formula Program named with $5B allocation; (6) contrarian source—Great Plains Institute 'nearly all DCFC scenarios lose money' fully cited and live; (7) hardware vs. install cost split clearly broken out. All six citations fetch live, every quoted string matches exactly, and all citations support the claims they are attached to (6/6 verified). Item 5 (Named operator) fails the rubric's specific requirement: the rubric demands GAAP net income/loss from a 10-K or 10-Q, and the citation must be the SEC filing itself—not a press release. The brief cites EVgo's FY2024 GAAP net loss of $126.7M, but the citation [6] is an 8-K EX-99.1 (earnings press release exhibit), explicitly labelled '(8-K EX-99.1)' in the Sources section. The rubric specifically excludes press releases, and an 8-K EX-99.1 is precisely that—a press release filed with the SEC—not a 10-K or 10-Q annual/quarterly report.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
writer · revision 2
searched/fetched 7× · wrote brief.md (7,989 chars)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
grader · pass 2  ✓ satisfied
All 7 coverage items are fully met: (1) Capex range gives hardware $45K-$100K+ and installation $40K-$150K+ per charger, plus NEVI average $915,420/station. (2) Demand charges quantified at $20/kW (explicit $/kW figure) with $9,600/month example. (3) Utilization breakeven at 20% or $0.53/kWh threshold explicitly modeled. (4) NEVI and IRA Section 30C named as subsidy programs. (5) EVgo GAAP net loss of $126.7M for FY2024 cited from the SEC filing itself (sec.gov XBRL R4.htm), not a press release. (6) Great Plains Institute contrarian source cited with thesis that 'nearly all DCFC scenarios lose money.' (7) Hardware vs. install cost split explicitly broken out. All 6 citations are LIVE (URLs return readable pages), all quoted strings appear verbatim in the fetched pages, and all support the claims they are cited on.

done: satisfied after 3 passes · 12m 56s

刚才发生了什么

循环运行了三轮评分。

第一稿覆盖了七项中的五项。评分器因两个遗漏将其退回：需求电费用定性描述而没有美元/kW 数字，以及指定运营商部分引用了第三方新闻文章而不是 SEC 文件。写作者添加了 $20/kW 数字并去 sec.gov 获取 EVgo 的 FY2024 净亏损。

第二轮通过了七项中的六项。写作者引用的 sec.gov 文件是 8-K Exhibit 99.1，即提交的收益新闻稿，而不是评分标准要求的 10-K 或 10-Q。评分器阅读了 URL，将其识别为新闻稿附件，并正是因为这个区别拒绝了它。第三轮，写作者在 EDGAR 上找到了 EVgo 的实际 10-K，评分器通过了它：7/7 覆盖、6/6 引用有效、引文匹配且支持声明。

两次拒绝都因为评分标准画了一条评分器可以检查的线。“$ /kW 数字或运营成本的百分比”和”SEC 文件本身，而不是新闻稿”是评分标准作者做出的决定；任务本身（“覆盖需求电费”、“指定运营商经济学”）会让两者都通过。第二次拒绝值得研究。写作者找到了一个 sec.gov URL，表面上满足了任务，但评分器仍然拒绝了它，因为评分标准区分了 8-K 新闻稿附件和 10-K。没有这个区别，循环会提前一轮结束，留下更差的简报。

注意上面的追踪来自原始 cookbook 运行。如果你自己运行，结果可能会有所不同。

5. 读取最终简报

最后，让我们拉取写作者生成的最后一个版本的 brief.md 并查看它覆盖了什么以及引用了什么。如果你想阅读完整的文本，请打印 content 本身。

# Reconstruct the final brief from the event log: full content on `write`,
# then apply each `edit` (old_string -> new_string) in order.
content = ""
for ev in client.beta.sessions.events.list(session.id, limit=1000, betas=BETAS):
    if ev.type != "agent.tool_use" or "brief.md" not in str(ev.input.get("file_path", "")):
        continue
    if ev.name == "write":
        content = ev.input["content"]
    elif ev.name == "edit":
        content = content.replace(ev.input["old_string"], ev.input["new_string"], 1)

# Show the structure and sources rather than the full prose.
for line in content.splitlines():
    if line.startswith(("#", "[")):
        print(line)

# Unit Economics of Public DC Fast Charging in the United States
## 1. Capex Range
## 2. Demand Charges
## 3. Utilization Breakeven
## 4. Subsidy Programs
## 5. Named-Operator Economics
## 6. Contrarian / Skeptical View
## 7. Hardware vs. Installation Cost Split
## Sources
[1] "A 150 to 350kW DCFC charging unit can cost anywhere from $45,000 to over $100,000, and installation costs can range from $40,000 to over $150,000." - Can public EV fast-charging stations be profitable in the United States? - https://www.mckinsey.com/features/mckinsey-center-for-future-mobility/our-insights/can-public-ev-fast-charging-stations-be-profitable-in-the-united-states
[2] "Waiving demand charges for fast chargers will shift costs to all ratepayers, including non-EV drivers." - Fast charging, high costs: Eliminating demand charges won't solve the problem - https://www.utilitydive.com/news/eliminating-demand-charges-wont-solve-EV-station-problems/689395/
[3] "the average total NEVI project cost is $915,420. The median total cost is $802,267" - NEVI DC Fast Charging Station Total Project Cost Averages $915,000 - https://www.paren.app/blog/nevi-dc-fast-charging-station-total-project-cost-averages-915-000
[4] "Today's economics and the average electric utility rates mean that nearly all DCFC scenarios lose money" - 'Nearly all' high voltage EV charging stations lose money: Report - https://www.utilitydive.com/news/nearly-all-high-voltage-ev-charging-stations-lose-money-report/561026/
[5] "apportions a total of $5 billion to States, D.C., and Puerto Rico over five years, from Fiscal Year 2022 through 2026" - Federal Funding Programs - https://www.transportation.gov/rural/ev/toolkit/ev-infrastructure-funding-and-financing/federal-funding-programs
[6] "Consolidated Statements of Operations - USD ($)$ in Thousands | 12 Months Ended Dec. 31, 2024 | Dec. 31, 2023" - EVgo Inc. Form 10-K, Consolidated Statements of Operations (XBRL), Period: 2024-12-31 - https://www.sec.gov/Archives/edgar/data/1821159/000155837025002400/R4.htm

你学到了什么

当你能写下好的标准时，Outcomes 适用。 注意细节和穷尽覆盖的任务是最明确的匹配；如果评分标准确定了标准，主观质量也可以。
评分标准必须让评分器产生证据。 没有这一点，它会批准展示给它的任何东西。
评分器是独立且无状态的。 它在自己的上下文窗口中运行，因此写作者无法说服它任何事情，并且每次迭代都会重新检查整个产物。
一次一个 outcome，但你可以链式使用。 一旦循环终止，session 再次变为对话模式，下一个 user.define_outcome 开始新的循环。

更多信息，请参阅 Outcomes 文档。

Tip

中文读者提示：本章节的代码和输出为英文原文。如需理解具体实现细节，可参考上方中文导读和代码注释。如有疑问，欢迎在 GitHub Issues 讨论。