文本摘要

使用 Claude 进行高效文本摘要的方法

🌐 查看英文原文 | 源码 Notebook

---

使用 Claude 进行摘要

简介

摘要是在自然语言处理中的一项关键任务，它涉及将大量文本压缩成更短、更易于消化的格式，同时保留关键信息。在当今信息丰富的世界中，能够快速从长篇文档中提取和综合要点对于各个行业和应用来说都非常有价值。

本指南侧重于利用 Claude 的摘要能力，特别强调法律文件。法律文件往往篇幅冗长且难以阅读——尤其是在有很多细小条款和法律术语的情况下。我们将探讨有效摘要此类文档的技术，评估摘要质量的方法，以及系统性地提高摘要性能的策略。

我们将涵盖的关键方面包括： - 设计有效的摘要提示 - 从文档中提取特定元数据 - 处理超出典型令牌限制的长文档 - 使用自动化方法（例如 ROUGE 分数和 Promptfoo 自定义方法）评估摘要质量 - 逐步提高摘要性能 - 有关如何优化您的摘要工作流程的一般性结论性提示

在本指南结束时，您将深入了解如何使用 Claude 实施和改进摘要任务，以及将这些技术应用于您自己的特定用例的框架。

---

请注意，以下代码示例中的变量名和函数名将保持不变。

def summarize_text(text, max_tokens=150):
    # 实现摘要逻辑
    pass

在开始之前，值得讨论一下本指南中的评估部分。评估摘要质量是一项众所周知具有挑战性的任务。与许多其他自然语言处理任务不同，摘要评估通常缺乏明确、客观的指标。这个过程可能非常主观，不同的读者可能会重视摘要的不同方面。传统的经验方法，如ROUGE分数，虽然有用，但在捕捉细微方面（如连贯性、事实准确性和相关性）方面存在局限性。此外，“最佳”摘要可能因具体用例、目标受众和所需细节水平的不同而有所差异。尽管存在这些挑战，但我们在本指南中探讨了多种不同的方法，这些方法可以结合使用，包括自动指标、正则表达式和特定任务的准则。在本指南中，我们认识到，最有效的方法通常涉及针对特定摘要任务量身定制的技术组合。

目录

1. 设置
2. 数据准备
3. 基本摘要
4. 多轮基本摘要
5. 高级技术
   • 引导摘要
   • 特定领域引导摘要
   • 元摘要
6. 摘要索引文档：高级RAG方法
   • 摘要RAG的最佳实践
7. 评估
8. 迭代改进
9. 结论和最佳实践

环境配置

要完成本指南，您需要安装以下包： - anthropic - pypdf - pandas - matplotlib - sklearn - numpy - rouge-score - nltk - seaborn - promptfoo (用于评估)

您还需要一个Claude API密钥。

让我们从安装所需的包和设置我们的环境开始：

# 安装所需的包
pip install requests
pip install flask

-- 创建数据库表
CREATE TABLE users (
    id INT PRIMARY KEY,
    username VARCHAR(50) NOT NULL,
    password VARCHAR(50) NOT NULL
);

{
    "name": "John Doe",
    "age": 30,
    "email": "john.doe@example.com"
}

请注意，以下代码块是示例，实际使用时请根据您的需求进行调整。

# 示例函数
def greet(name):
    return f"Hello, {name}!"

# 示例类
class User:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def display_info(self):
        print(f"Name: {self.name}, Age: {self.age}")

# 示例RAG模型
class RAG:
    def __init__(self, model_name, tokenizer, device):
        self.model_name = model_name
        self.tokenizer = tokenizer
        self.device = device

    def embed(self, text):
        # 将文本转换为嵌入表示
        pass

    def generate_prompt(self, text):
        # 生成提示
        pass

    def generate_response(self, prompt):
        # 生成响应
        pass
``````python
# install packages
!pip install anthropic pypdf pandas matplotlib numpy rouge-score nltk seaborn --quiet

import re

import anthropic
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

# Set up Anthropic client
# You can set up a .env file with your API key to keep it private, and import it like so:
# from dotenv import load_dotenv
# load_dotenv()

# or add your key directly
api_key = "ANTHROPIC_API_KEY"  # Replace ANTHROPIC_API_KEY with your actual API key
client = anthropic.Anthropic(api_key=api_key)

print("Setup complete!")

Setup complete!
```## 数据准备
在开始总结文档之前，我们需要准备我们的数据。这包括从PDF中提取文本、清理文本，并确保其可以输入到我们的语言模型中。为了演示目的，我们从sec.gov网站获取了一份[公开的转租协议](https://www.sec.gov/Archives/edgar/data/1045425/000119312507044370/dex1032.htm)。

如果您有任何想要测试的PDF文件，请将其导入此目录，然后更改下面的文件路径。**如果您只想通过复制粘贴使用文本块，请跳过此步骤并定义 `text = <text blob>`**。

以下是一组处理此过程的函数：```python
import re

import pypdf

pdf_path = "data/Sample Sublease Agreement.pdf"


def extract_text_from_pdf(pdf_path):
    with open(pdf_path, "rb") as file:
        reader = pypdf.PdfReader(file)
        text = ""
        for page in reader.pages:
            text += page.extract_text() + "\n"
    return text


def clean_text(text):
    # Remove extra whitespace
    text = re.sub(r"\s+", " ", text)
    # Remove page numbers
    text = re.sub(r"\n\s*\d+\s*\n", "\n", text)
    return text.strip()


def prepare_for_llm(text, max_tokens=180000):
    # Truncate text to fit within token limit (approximate)
    return text[: max_tokens * 4]  # Assuming average of 4 characters per token


def get_llm_text(path):
    extracted_text = extract_text_from_pdf(path)
    cleaned_text = clean_text(extracted_text)
    llm_ready_text = prepare_for_llm(cleaned_text)
    return llm_ready_text


# You can now use get_llm_text in your LLM prompt
text = get_llm_text(pdf_path)
print(text[:500])

EX-10.32 7 dex1032.htm SUBLEASE AGREEMENT Exhibit 10.32 SUBLEASE AGREEMENT THIS SUBLEASE AGREEMENT (“Sublease ”), is dated as of April 1, 2006, by and between COHEN BROTHERS, LLC d/b/a COHEN & COMP ANY (“Sublessor ”) and TABERNA CAPIT AL MANAGEMENT , LLC (“Sublessee ”), collectively , the “ Parties ” and each a “ Party ”. WHEREAS, Sublessor is the lessee under a written lease agreement dated June 22, 2005 wherein Brandywine Cira, L.P ., a Delaware limited partnership (“ Lessor ”), leased Suite N
```这个设置使我们能够轻松处理PDF文档并为摘要做准备。在下一节中，我们将从基本的摘要方法开始，然后在此基础上构建更高级的技术。

## 基本摘要

让我们从使用Claude的简单摘要函数开始。这是使用Claude总结上述文档文本的简单尝试。随着我们通过本指南的进展，我们将改进这种方法。

有一点需要注意，虽然这看起来很基础，但我们实际上已经在使用Claude的一些重要功能了。值得注意的一点是助手角色和停止序列的使用。助手前言将Claude设置好，以便在最终短语`<summary>`之后直接包含摘要。停止序列`</summary>`然后告诉Claude停止生成。这是我们将在整个指南中继续使用的模式。

```python
def summarize_text(text):
    # 代码实现摘要逻辑
    pass

{
    "summary": "这里是摘要内容"
}

SELECT * FROM documents WHERE id = 1;

请注意，虽然这个方法可能看起来很基础，但我们实际上已经在使用Claude的一些重要功能了。 :::```python def basic_summarize(text, max_tokens=1000): # Prompt the model to summarize the text prompt = f”““Summarize the following text in bullet points. Focus on the main ideas and key details: {text}”“”

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=max_tokens,
    system="You are a legal analyst known for highly accurate and detailed summaries of legal documents.",
    messages=[
        {"role": "user", "content": prompt},
        {
            "role": "assistant",
            "content": "Here is the summary of the legal document: <summary>",
        },
    ],
    stop_sequences=["</summary>"],
)

return response.content[0].text

```python
basic_response = basic_summarize(text, max_tokens=1000)

print(basic_response)

Key Points:
•Between parties: COHEN BROTHERS, LLC d/b/a COHEN & COMPANY (Sublessor) and TABERNA CAPITAL MANAGEMENT, LLC (Sublessee).
•Signed on April 1, 2006.
•Premises: 2,000 square feet of office space in Suite 1703 in the Cira Center at 2929 Arch Street, Philadelphia.

Major Terms:
•Term: 5 years starting April 1, 2006
•Payment: Fixed rent increases annually from $34.50/sf to $37.34/sf over the term.
•Utilities: Tenant pays for electricity and pro rata share of building expenses.
•Use: General office use only.
•Assignment/Subletting: Requires prior written consent of landlord.

Key Obligations:
•Tenant must maintain insurance including liability and property insurance.  
•Tenant responsible for interior maintenance/condition of the premises.
•Tenant must comply with all building rules and regulations.
•Tenant must maintain premises in good order and repair.

Notable Provisions:
•Sublessor can recapture premises if tenant tries to assign/sublet without proper consent.
•Default provisions give sublessor multiple remedies including termination and accelerated rent.  
•Tenant must indemnify landlord for claims related to tenant's use or actions.
•Tenant responsible for maintaining and repairing interior of premises.

This appears to be a fairly standard commercial office sublease with typical provisions regarding tenant obligations, insurance requirements, default remedies, etc. The sublessor retains significant control and remedies while the sublessee has standard obligations for an office tenant.
```This basic approach provides a simple summary, but it may not capture all the nuances we need for legal or financial documents. As you'll notice too, when you rerun the cell above, there is no standard, formalized output. Instead, we retrieve a basic summarization of the document, without much structured output to parse through. This makes it harder to read, more difficult to trust (how do we know it didn't miss something?) and thus trickier to use in any real world context.

Let's see if we can tweak our prompt to get a more structure version of our summary.

## 多次基本摘要

It's pretty cool that we can summarize massive documents so fast, but we can do even better. Let's try adding a few examples to our prompt to see if we can improve the output and create a bit of structure in our output before we move on to even more advanced techniques. 

Note, here we haven't really change the actual format of the request, although we have appended 2 additional pieces: 

1. We've told the model "do not preamble". This can often be a good idea when it comes to constraining the model output to just the answer we want, without that initial form conversational angle you might be familiar with if you've used Claude before. It's particularly important when we aren't using other "instructions" within the prompt (as we might do later in this guide).
2. We append 3 examples of summarized documents. This is called few-shot or multi-shot learning, and it can help the model understand what we're looking for.

Let's see how the output changes:

```python
# Example code
def summarize(text):
    # Code to summarize the text
    pass

{
    "summary": "This is a summary of the document."
}

SELECT * FROM documents WHERE id = 1;

请注意，以上代码仅为示例，实际应用中可能需要根据具体情况进行调整。 :::```python # We import from our data directory to save space in our notebook from data.multiple_subleases import document1, document2, document3, sample1, sample2, sample3

def basic_summarize_multishot(text, max_tokens=1000): # Prompt the model to summarize the text prompt = f”““Summarize the following text in bullet points. Focus on the main ideas and key details: {text}

Do not preamble.

Use these examples for guidance in summarizing:

<example1>
    <original1>
        {document1}
    </original1>

    <summary1>
        {sample1}
    </summary1>
</example1>

<example2>
    <original2>
        {document2}
    </original2>

    <summary2>
        {sample2}
    </summary2>
</example2>

<example3>
    <original3>
        {document3}
    </original3>

    <summary3>
        {sample3}
    </summary3>
</example3>
"""

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=max_tokens,
    system="You are a legal analyst known for highly accurate and detailed summaries of legal documents.",
    messages=[
        {"role": "user", "content": prompt},
        {
            "role": "assistant",
            "content": "Here is the summary of the legal document: <summary>",
        },
    ],
    stop_sequences=["</summary>"],
)

return response.content[0].text

```python
basic_multishot_response = basic_summarize_multishot(text, max_tokens=1000)

print(basic_multishot_response)

Description: This is a sublease agreement between Cohen Brothers, LLC (Sublessor) and Taberna Capital Management, LLC (Sublessee) for office space in Philadelphia.

<parties involved>
Sublessor: Cohen Brothers, LLC d/b/a Cohen & Company
Sublessee: Taberna Capital Management, LLC
Original lessor: Brandywine Cira, L.P.
</parties involved>

<property details> 
Address: 2929 Arch Street, Suite 1703, Philadelphia, PA
Description: 2,000 square feet of office space with access to file space, printers, copiers, kitchen, conference rooms
Permitted use: General office use
</property details>

<term and rent>
Start date: April 1, 2006
End date: 5 years from start date
Monthly rent:
• Months 1-12: $5,750
• Months 13-24: $5,865
• Months 25-36: $5,981.67
• Months 37-48: $6,101.67
• Months 49-60: $6,223.33
</term and rent>

<responsibilities>
Utilities: Not explicitly specified
Maintenance: Not explicitly specified
Repairs: Tenant responsible for damage repairs
Insurance: Tenant required to maintain liability insurance with $3M limit and workers compensation insurance
</responsibilities>

<special provisions>
Default: Detailed events of default and remedies specified
Holdover: Double rent for unauthorized holdover period
Assignment/Subletting: Not permitted without landlord consent
Alterations: Require landlord consent
Access to services: Includes file space, copiers, conference rooms, receptionist services
</special provisions>
```如果您查看我们提供的示例，您会看到上面的输出格式与示例中的格式相同（*前往data/<任意.txt文件>查看*）。这很有趣——我们没有明确告诉Claude要遵循示例的格式，但它似乎还是注意到了这一点。这说明了少量样本学习的力量，以及Claude如何从几个示例中推广到新的输入。

## 高级摘要技术

### 引导式摘要

引导式摘要是指我们明确为模型定义一个框架，使其在摘要任务中遵守。我们可以通过提示来实现这一点，改变提示的细节来引导Claude更加或较少详细，包含更多或更少的术语，或者提供更高或更低级别的上下文摘要。对于法律文件，我们可以引导摘要专注于特定方面。

注意，我们很可能通过示例（我们上面已经探索过）来实现我们下面展示的相同格式化输出！```python
def guided_legal_summary(text, max_tokens=1000):
    # Prompt the model to summarize the text
    prompt = f"""Summarize the following legal document. Focus on these key aspects:

    1. Parties involved
    2. Main subject matter
    3. Key terms and conditions
    4. Important dates or deadlines
    5. Any unusual or notable clauses

    Provide the summary in bullet points under each category.

    Document text:
    {text}

    """

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=max_tokens,
        system="You are a legal analyst known for highly accurate and detailed summaries of legal documents.",
        messages=[
            {"role": "user", "content": prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the legal document: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text

# Example usage
legal_summary = guided_legal_summary(text)
print(legal_summary)

1. Parties Involved
- Sublessor: Cohen Brothers, LLC d/b/a Cohen & Company
- Sublessee: Taberna Capital Management, LLC
- Original Landlord: Brandywine Cira, L.P. (Master Lease landlord)

2. Main Subject Matter
- Sublease agreement for Suite 1703 at Cira Centre, 2929 Arch Street, Philadelphia, PA
- 2,000 square feet of office space within the Master Premises of 13,777 rentable square feet
- Includes furniture, file space, printers, copiers, kitchen, conference room facilities and receptionist/secretarial services

3. Key Terms and Conditions
- Initial Term: 5 years from April 1, 2006 
- Fixed Rent: Escalating annual rent schedule starting at $34.50/sq ft in Year 1 ($69,000 annually) up to $37.34/sq ft in Year 5 ($74,680 annually)
- Pro rata share of operating expenses and utilities
- No assignment or subletting without Sublessor's prior written consent
- Sublessee takes premises "AS IS"
- Sublessee must maintain required insurance coverage
- Default provisions for non-payment, breach of lease terms, bankruptcy, etc.

4. Important Dates/Deadlines  
- Commencement Date: April 1, 2006
- Expiration Date: 5 years from Commencement Date
- Fixed Rent payable monthly in advance on 1st of each month
- 5-day grace period for late payments before default

5. Notable Clauses
- Indemnification requirements for both parties
- Holdover rent at 2x monthly rate if Sublessee remains after term ends
- Sublessor not liable for utilities/services interruption
- Sublessee responsible for any construction liens
- Confession of judgment provision
- Waiver of jury trial provision
```这无疑使得解析文档中最相关的部分和理解特定行项及重要条款的含义变得更加容易。

### 领域特定的引导式摘要

您可以将上述引导式摘要提示应用于任何类型的文档，但我们可以通过针对特定文档类型进行定制来使其更加强大。例如，如果我们知道我们正在处理一个分租协议，我们可以引导模型关注该特定类型文档中最相关的法律术语和概念。这在我们使用Claude处理特定用例并明确知道我们想要提取的最相关值时最为相关。

以下是我们可能修改用于分租协议的引导式摘要函数的示例。请注意，我们还将添加'model'作为函数的附加参数，这样我们可以根据任务更轻松地选择不同的摘要模型：

```html
::: callout-notice
Here's an example of how we might modify our guided summarization function for sublease agreements. Note that we'll also add the 'model' as an additional parameter to our function so that we can more easily choose different models for summarization based upon the task:
:::

```python
# Example of modifying guided summarization function for sublease agreements
def guided_summarization_for_sublease(model, document):
    # Implementation details here
    pass
``````python
def guided_sublease_summary(text, model="claude-sonnet-4-6", max_tokens=1000):
    # Prompt the model to summarize the sublease agreement
    prompt = f"""Summarize the following sublease agreement. Focus on these key aspects:

    1. Parties involved (sublessor, sublessee, original lessor)
    2. Property details (address, description, permitted use)
    3. Term and rent (start date, end date, monthly rent, security deposit)
    4. Responsibilities (utilities, maintenance, repairs)
    5. Consent and notices (landlord's consent, notice requirements)
    6. Special provisions (furniture, parking, subletting restrictions)

    Provide the summary in bullet points nested within the XML header for each section. For example:

    <parties involved>
    - Sublessor: [Name]
    // Add more details as needed
    </parties involved>

    If any information is not explicitly stated in the document, note it as "Not specified". Do not preamble.

    Sublease agreement text:
    {text}

    """

    response = client.messages.create(
        model=model,
        max_tokens=max_tokens,
        system="You are a legal analyst specializing in real estate law, known for highly accurate and detailed summaries of sublease agreements.",
        messages=[
            {"role": "user", "content": prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the sublease agreement: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text

# Example usage
sublease_summary = guided_sublease_summary(text)
print(sublease_summary)

<parties involved>
- Sublessor: Cohen Brothers, LLC (d/b/a Cohen & Company) 
- Sublessee: Taberna Capital Management, LLC
- Original Lessor: Brandywine Cira, L.P. (Master Lease holder)
</parties involved>

<property details>
- Address: 2929 Arch Street, Suite 1703, Philadelphia, PA
- Description: 2,000 square feet of office space in Suite 1703 
- Permitted Use: General office use
- Includes: Access to file space, printers, copiers, kitchen, conference rooms, receptionist/secretarial services
</property details>

<term and rent>
- Start Date: April 1, 2006
- End Date: 5 years from commencement
- Monthly Rent: Escalating schedule starting at $5,750 in year 1 up to $6,223.33 in year 5
- Security Deposit: Not specified
</term and rent>

<responsibilities>
- Utilities: Sublessee pays proportional share of utilities and operating expenses
- Maintenance: Sublessor responsible for base building maintenance
- Repairs: Sublessee responsible for repairs due to its use
- Insurance: Sublessee must maintain general liability and property insurance
</responsibilities>

<consent and notices>
- Landlord's Consent: Required for assignment/subletting
- Notice Requirements: All notices must be in writing and delivered to specified addresses
- Sublessor's Consent: Required for alterations, improvements, signage
</consent and notices>

<special provisions>
- Furniture: Included in lease
- Parking: Not included
- Assignment: No assignment/subletting without Sublessor's consent
- Default Remedies: Specified remedies including termination and accelerated rent
</special provisions>
```由于我们决定使用XML标签输出总结的每个部分，现在我们可以像这样单独解析它们（这也可以通过JSON或其他任何格式完成）：

```markdown
Because we decided to output each section of the summary in XML tags, we can now parse them individually out like so (this could also be done via JSON or any other format):

```python
def parse_summary(summary_xml):
    # Parse the XML and extract individual sections
    pass

::: callout-notice
请注意，这里的XML格式是示例性的，实际应用中可能需要根据具体情况进行调整。
:::
``````python
import re


def parse_sections_regex(text):
    pattern = r"<(.*?)>(.*?)</\1>"
    matches = re.findall(pattern, text, re.DOTALL)

    parsed_sections = {}
    for tag, content in matches:
        items = [item.strip("- ").strip() for item in content.strip().split("\n") if item.strip()]
        parsed_sections[tag] = items

    return parsed_sections


# Parse the sections
parsed_sections = parse_sections_regex(sublease_summary)

# Check if parsing was successful
if isinstance(parsed_sections, dict) and "parties involved" in parsed_sections:
    print("Parties involved:")
    for item in parsed_sections["parties involved"]:
        print(f"- {item}")
else:
    print("Error: Parsing failed or 'parties involved' section not found.")
    print("Parsed result:", parsed_sections)

Parties involved:
- Sublessor: Cohen Brothers, LLC (d/b/a Cohen & Company)
- Sublessee: Taberna Capital Management, LLC
- Original Lessor: Brandywine Cira, L.P. (Master Lease holder)
```### 包含多文档上下文（元摘要）

如果我们有很多与同一客户相关的文档怎么办？我们可以使用分块方法来处理这个问题。这是一种将文档分解成更小、更易于管理的块，然后分别处理每个块的技术。然后我们可以将每个块的摘要组合起来，创建整个文档的元摘要。当我们想要总结大量文档或想要总结一个非常长的单个文档时，这尤其有帮助。

以下是我们可能如何进行此操作的示例：```python
from data.multiple_subleases import document1, document2, document3


def chunk_text(text, chunk_size=2000):
    return [text[i : i + chunk_size] for i in range(0, len(text), chunk_size)]


def summarize_long_document(text, max_tokens=2000):
    chunks = chunk_text(text)

    # Iterate over chunks and summarize each one
    # We use guided_legal_summary here, but you can use basic_summarize or any other summarization function
    # Note that we'll also use haiku for the interim summaries, and the 3.5 sonnet for the final summary
    chunk_summaries = [
        guided_sublease_summary(chunk, model="claude-haiku-4-5", max_tokens=max_tokens)
        for chunk in chunks
    ]

    final_summary_prompt = f"""

    You are looking at the chunked summaries of multiple documents that are all related. Combine the following summaries of the document from different truthful sources into a coherent overall summary:

    {"".join(chunk_summaries)}

    1. Parties involved (sublessor, sublessee, original lessor)
    2. Property details (address, description, permitted use)
    3. Term and rent (start date, end date, monthly rent, security deposit)
    4. Responsibilities (utilities, maintenance, repairs)
    5. Consent and notices (landlord's consent, notice requirements)
    6. Special provisions (furniture, parking, subletting restrictions)

    Provide the summary in bullet points nested within the XML header for each section. For example:

    <parties involved>
    - Sublessor: [Name]
    // Add more details as needed
    </parties involved>

    If any information is not explicitly stated in the document, note it as "Not specified".

    Summary:
    """

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=max_tokens,
        system="You are a legal expert that summarizes notes on one document.",
        messages=[
            {"role": "user", "content": final_summary_prompt},
            {
                "role": "assistant",
                "content": "Here is the summary of the legal document: <summary>",
            },
        ],
        stop_sequences=["</summary>"],
    )

    return response.content[0].text


# Example usage
# combine 3 documents (all related) together
text = document1 + document2 + document3
long_summary = summarize_long_document(text)
print(long_summary)

<parties involved>
- Sublessor: Apex Innovations, Inc. (Delaware corporation) later identified as TechHub Enterprises, LLC
- Sublessee: NanoSphere Solutions, Inc. and Quantum Dynamics, LLC (California LLC)
- Original Lessor: Innovate Properties, LLP
</parties involved>

<property details>
- Address: 9876 Innovation Park, Building C, San Francisco, CA 94107
- Description: Approximately 15,000-25,000 square feet of office and laboratory space
- Permitted Use: General office purposes, research and development, and laboratory uses consistent with BSL-2 facility requirements
</property details>

<term and rent>
- Start Date: September 1, 2023
- End Date: August 31, 2026 (with option to extend for 3-5 additional years)
- Monthly Rent: Starting at $75,000/month with annual 3% increases
- Security Deposit: $450,000-$787,500
</term and rent>

<responsibilities>
- Utilities: Sublessee responsible for all utilities and services, including electricity, gas, water, sewer, telephone, internet, and janitorial
- Maintenance: Sublessee responsible for interior maintenance, repairs and replacements, including walls, floors, ceilings, doors, windows, fixtures
- Repairs: Sublessee responsible for repairs except building structure, exterior walls, roof which are Sublessor's responsibility
</responsibilities>

<consent and notices>
- Landlord's Consent: Required for assignments, subletting, alterations
- Notice Requirements: 30 days written notice for defaults, insurance changes; 9-12 months notice for term extensions
</consent and notices>

<special provisions>
- Furniture: Right to install furniture/equipment 15-30 days before commencement
- Parking: Non-exclusive right to use common parking facilities
- Subletting Restrictions: No assignment/subletting without prior written consent, except to affiliated entities
- Additional: Hazardous materials restrictions, OFAC compliance requirements, jury trial waiver
</special provisions>
```## 摘要索引文档：一种高级 RAG 方法

Summary Indexed Documents 是一种在文档级别操作的高级检索增强生成（RAG）方法。

这种方法相较于传统的 RAG 技术具有多个优势，尤其是在涉及大文档或精确信息检索至关重要的场景中。

#### 工作原理

1. 文档摘要：为语料库中的每个文档生成简洁的摘要（查询的文本子集被快速查询和摘要）。
2. 上下文窗口优化：确保所有摘要都适合语言模型的上下文窗口。
3. 相关性评分：要求模型对每个摘要与所提查询的相关性进行排序。
4. 重新排序（可选）：应用重新排序技术进一步优化和压缩前 K 个结果。
5. 回答当前查询。

这种方法具有一些明显的优势：
- 比传统 RAG 方法更高效地排序文档以进行检索，使用更少的上下文。
- 在特定任务上表现优异：优于其他 RAG 方法，始终将正确的文档排在第一位。
- 优化信息检索：重新排序有助于压缩结果，确保模型呈现最简洁和相关的信息。

```python
def generate_summary(text):
    # 生成摘要的代码
    pass

def optimize_context_window(summaries, model_context_window):
    # 优化上下文窗口的代码
    pass

def score_relevance(summaries, query):
    # 评分相关性的代码
    pass

def rerank_results(results, reranking_technique):
    # 重新排序结果的代码
    pass

def answer_query(query):
    # 回答查询的代码
    pass
``````python
class LegalSummaryIndexedDocuments:
    def __init__(self, client):
        self.client = client  # Claude client
        self.documents: list[dict[str, str]] = []  # List of docs to store
        self.summaries: list[str] = []

    def add_document(self, doc_id: str, content: str):
        # Adds a document to the index
        self.documents.append({"id": doc_id, "content": content})

    def generate_summaries(self):
        # Generates summaries for all documents in the index
        for doc in self.documents:
            summary = self._generate_legal_summary(doc["content"])
            self.summaries.append(summary)

    def _generate_legal_summary(self, content: str) -> str:
        # Note how we constrain the content to a maximum of 2000 words. We do this because we don't need that much information for the intial ranking.
        prompt = f"""
        Summarize the following sublease agreement. Focus on these key aspects:

        1. Parties involved (sublessor, sublessee, original lessor)
        2. Property details (address, description, permitted use)
        3. Term and rent (start date, end date, monthly rent, security deposit)
        4. Responsibilities (utilities, maintenance, repairs)
        5. Consent and notices (landlord's consent, notice requirements)
        6. Special provisions (furniture, parking, subletting restrictions)

        Provide the summary in bullet points nested within the XML header for each section. For example:

        <parties involved>
        - Sublessor: [Name]
        // Add more details as needed
        </parties involved>

        If any information is not explicitly stated in the document, note it as "Not specified".

        Sublease agreement text:
        {content[:2000]}...

        Summary:
        """

        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=500,
            temperature=0.2,
            messages=[
                {"role": "user", "content": prompt},
                {
                    "role": "assistant",
                    "content": "Here is the summary of the legal document: <summary>",
                },
            ],
            stop_sequences=["</summary>"],
        )
        return response.content[0].text

    def rank_documents(self, query: str, top_k: int = 3) -> list[tuple[str, float]]:
        """
        Rank documents based on their relevance to the given query.
        We use Haiku here as a cheaper, faster model for ranking.
        """
        ranked_scores = []
        for summary in self.summaries:
            prompt = f"Legal document summary: {summary}\n\nLegal query: {query}\n\nRate the relevance of this legal document to the query on a scale of 0 to 10. Only output the numeric value:"

            response = client.messages.create(
                model="claude-haiku-4-5",
                max_tokens=2,
                temperature=0,
                messages=[{"role": "user", "content": prompt}],
            )
            ranked_score = float(response.content[0].text)
            ranked_scores.append(ranked_score)

        ranked_indices = np.argsort(ranked_scores)[::-1][:top_k]
        return [(self.documents[i]["id"], ranked_scores[i]) for i in ranked_indices]

    def extract_relevant_clauses(self, doc_id: str, query: str) -> list[str]:
        """
        Extracts relevant clauses from a document based on a query.
        """
        doc_content = next(doc["content"] for doc in self.documents if doc["id"] == doc_id)

        prompt = f"""
        Given the following legal query and document content, extract the most relevant clauses or sections and write the answer to the query.
        Provide each relevant clause or section separately, preserving the original legal language:

        Legal query: {query}

        Document content: {doc_content}...

        Relevant clauses or sections (separated by '---'):"""

        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1000,
            temperature=0,
            messages=[{"role": "user", "content": prompt}],
        )

        clauses = re.split(r"\n\s*---\s*\n", response.content[0].text.strip())
        return [clause.strip() for clause in clauses if clause.strip()]

from data.multiple_subleases import document1, document2, document3

lsid = LegalSummaryIndexedDocuments(client=client)

# Add documents
lsid.add_document("doc1", document1)
lsid.add_document("doc2", document2)
lsid.add_document("doc3", document3)

# Generate summaries - this would happen at ingestion
lsid.generate_summaries()

# Rank documents for a legal query
legal_query = "What contract is for the sublessor Apex Innovations, LLC?"
ranked_results = lsid.rank_documents(legal_query)

print("Initial ranking:", ranked_results)

# Extract relevant clauses from the top-ranked document
top_doc_id = ranked_results[0][0]
relevant_clauses = lsid.extract_relevant_clauses(top_doc_id, legal_query)

print("\nRelevant clauses from the top-ranked document:")
for i, clause in enumerate(relevant_clauses[1:], 1):
    print(f"Clause {i}: {clause}")

Initial ranking: [('doc1', 8.0), ('doc3', 0.0), ('doc2', 0.0)]

Relevant clauses from the top-ranked document:
Clause 1: COMMERCIAL SUBLEASE AGREEMENT

THIS COMMERCIAL SUBLEASE AGREEMENT (hereinafter referred to as the "Sublease") is made and entered into on this 15th day of August, 2023 (the "Effective Date"), by and between:

SUBLESSOR: Apex Innovations, Inc., a Delaware corporation with its principal place of business at 1234 Tech Boulevard, Suite 5000, San Francisco, CA 94105 (hereinafter referred to as the "Sublessor")
Clause 2: WHEREAS, Sublessor is the Tenant under that certain Master Lease Agreement dated January 1, 2020 (hereinafter referred to as the "Master Lease"), wherein Innovate Properties, LLP (hereinafter referred to as the "Master Lessor") leased to Sublessor those certain premises consisting of approximately 50,000 square feet of office space located at 9876 Innovation Park, Building C, Floors 10-12, San Francisco, CA 94107 (hereinafter referred to as the "Master Premises");
Clause 3: Answer: There appears to be an error in the legal query. The query refers to "Apex Innovations, LLC" but the document shows that the sublessor is actually "Apex Innovations, Inc.", a Delaware corporation. The contract for Apex Innovations, Inc. is:

1. A Commercial Sublease Agreement dated August 15, 2023, where they are the Sublessor
2. A Master Lease Agreement dated January 1, 2020, where they are the Tenant under Innovate Properties, LLP
```### 摘要 RAG 的最佳实践

- 最佳摘要长度：尝试不同的摘要长度，以找到简洁和信息性之间的最佳平衡。
- 迭代重排序：考虑进行多轮重排序以获得更精确的结果，尤其是在处理较大的文档集时。
- 缓存：实现摘要和初始排名的缓存机制，以提高重复查询的性能。

Summary Indexed Documents 提供了一种强大的 RAG 方法，尤其是在涉及大型文档或精确信息检索至关重要的场景中表现出色。通过利用文档摘要、对数概率评分和可选的重排序，这种方法为检索和向语言模型呈现相关信息提供了一种高效且有效的方式。

## 评估

如本食谱引言中所述，评估摘要的质量是一项艰巨的工作。这是因为有多种方式可以总结文档，不同的摘要可能同样有效。根据用例，摘要的不同方面可能更重要或更不重要。

您可以在[这里](https://docs.claude.com/en/docs/prompt-engineering)了解更多关于我们实证方法到提示工程的信息。使用 Jupyter Notebook 是开始提示工程的好方法，但随着您的数据集变大和提示增多，利用与您一起扩展的工具变得很重要。

在本指南的这一部分，我们将探讨使用 [Promptfoo](https://www.promptfoo.dev/)，一个开源的 LLM 评估工具包。要开始，请转到 `./evaluation` 目录并检出 `./evaluation/README.md`。

当您成功运行评估后，请返回此处查看结果。您还可以使用命令 `npx promptfoo@latest view` 以动态方式查看结果，在创建一些结果后。```python
import re

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

%matplotlib inline
plt.style.use("seaborn")

# Load the data
df = pd.read_csv("data/results.csv")


# Function to extract PASS/FAIL and score
def extract_result(text):
    match = re.search(r"\[(PASS|FAIL)\]\s*\((\d+\.\d+)\)", str(text))
    if match:
        return match.group(1), float(match.group(2))
    return "UNKNOWN", 0.0


# Apply the extraction to relevant columns
for col in df.columns[2:]:
    df[f"{col}_result"], df[f"{col}_score"] = zip(*df[col].apply(extract_result), strict=False)

# Prepare data for grouped accuracy score
models = ["3.5 Sonnet", "3.0 Haiku"]
prompts = ["basic_summarize", "guided_legal_summary", "summarize_long_document"]

results = []
for model in models:
    for prompt in prompts:
        col = f"[{model}] prompts.py:{prompt}_result"
        if col in df.columns:
            pass_rate = (df[col] == "PASS").mean()
            results.append({"Model": model, "Prompt": prompt, "Pass Rate": pass_rate})

result_df = pd.DataFrame(results)

# 1. Grouped bar chart for accuracy scores
plt.figure(figsize=(12, 6))
result_pivot = result_df.pivot(index="Prompt", columns="Model", values="Pass Rate")
result_pivot.plot(kind="bar")
plt.title("Pass Rate by Model and Prompt")
plt.ylabel("Pass Rate")
plt.legend(title="Model")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()

# 2. Box plot of scores
plt.figure(figsize=(8, 8))
score_cols = [col for col in df.columns if col.endswith("_score")]
score_data = df[score_cols].melt()
sns.boxplot(x="variable", y="value", data=score_data)
plt.title("Distribution of Scores")
plt.xticks(rotation=90)
plt.xlabel("Model and Prompt")
plt.ylabel("Score")
plt.tight_layout()
plt.show()

# Display summary statistics
summary_stats = df[[col for col in df.columns if col.endswith("_score")]].describe()
display(summary_stats)

/var/folders/c8/rjj6d5_15tj4qh_zhlnz9xxr0000gp/T/ipykernel_58010/2701104606.py:7: MatplotlibDeprecationWarning: The seaborn styles shipped by Matplotlib are deprecated since 3.6, as they no longer correspond to the styles shipped by seaborn. However, they will remain available as 'seaborn-v0_8-<style>'. Alternatively, directly use the seaborn API instead.
  plt.style.use('seaborn')

<Figure size 1200x600 with 0 Axes>

<Figure size 800x550 with 1 Axes>

<Figure size 800x800 with 1 Axes>

``` [3.0 Haiku] prompts.py:basic_summarize_score
count 9.000000
mean 1.423333
std 0.135647
min 1.190000
25% 1.400000
50% 1.440000
75% 1.510000
max 1.600000

   [3.0 Haiku] prompts.py:guided_legal_summary_score  \

count 9.000000
mean 1.443333
std 0.146969
min 1.270000
25% 1.300000
50% 1.450000
75% 1.490000
max 1.660000

   [3.0 Haiku] prompts.py:summarize_long_document_score  \

count 9.000000
mean 1.522222
std 0.092030
min 1.440000
25% 1.460000
50% 1.460000
75% 1.630000
max 1.660000

   [3.5 Sonnet] prompts.py:basic_summarize_score  \

count 9.000000
mean 1.088889
std 0.535547
min 0.000000
25% 1.210000
50% 1.290000
75% 1.400000
max 1.490000

   [3.5 Sonnet] prompts.py:guided_legal_summary_score  \

count 9.000000
mean 1.330000
std 0.086458
min 1.230000
25% 1.280000
50% 1.310000
75% 1.340000
max 1.480000

   [3.5 Sonnet] prompts.py:summarize_long_document_score  

count 9.000000
mean 1.475556
std 0.285750
min 0.750000
25% 1.450000
50% 1.600000
75% 1.640000
max 1.650000 ```查看结果，似乎我们的最佳表现者是3.5 Sonnet，在所有评估中通过率为66%，只有3次测试失败（45次测试中，如果某个测试因提示失败，则视为失败）。这只是开始——我们在这里使用的是完全虚构的数据，这些数据要么是（a）Claude生成的，要么是（b）从SEC政府网站上获取的。当我们拥有真实数据时，我们可以做得更好，因为我们对我们正在处理的独特问题集了解得更多。

迭代改进

当我们更深入地查看评估结果时，仍有改进的空间。这就是提示工程中迭代部分的作用所在。以下是我们可以采取的改进结果的步骤：

1. 分析Promptfoo的结果，以确定优势和劣势——例如，我们的contains评估似乎经常失败。这可能是由于一些文档中不包含XML标签中所需的信息。如果我们想准确评估性能，我们应该改进这个评估（但这是一个例子！）。
2. 精炼提示以解决特定问题（例如，提高简洁性或完整性）——我们看到多轮提示是一个很好的初次尝试。这是我们应该结合一些高级技术来进一步提高性能的事情。
3. 尝试不同的长文档分块策略。
4. 微调温度和max_tokens参数。
5. 实施后处理步骤以增强摘要质量。

结论和最佳实践

在这份指南中，我们介绍了使用Claude对文档进行摘要的各种技术，重点关注法律文件。构建一个完美的摘要系统和评估框架是一项艺术：它需要结合这些方法才能成功。正如我们一开始提到的，摘要是一个非常主观的话题，但我们已经找到了一些可行的方式来评估它，并对我们的结果感到满意。也要始终记住——你并不是在将你的结果与100%的准确性进行比较。你是在比较你自己完成这个复杂任务的能力；正如本指南中所展示的Claude的速度和效率，你开始意识到这种方法论方法的真实好处，因此你可以把时间花在真正的决策上。

总结一下这些建议，我们提供了一些需要记住的最佳实践：

1. 设计清晰和具体的提示。使用诸如“不要前言”之类的词语来限制输出。
2. 至少使用2个示例。
3. 对于特定领域的文档，使用引导式摘要。
4. 对于长文档，实施有效的、高级的策略。
5. 定期评估和改进你的方法。
6. 考虑AI生成摘要的伦理影响和局限性。

---

Tip

中文读者提示：本章节的代码和输出为英文原文。如需理解具体实现细节，可参考上方中文导读和代码注释。如有疑问，欢迎在 GitHub Issues 讨论。