Claude 3 RAG Agent

基于 Pinecone 的 Claude RAG Agent

🌐 查看英文原文 | 源码 Notebook

—python !python --version

Python 3.10.12
```# Claude 3 RAG Agents with LangChain v1

LangChain v1带来了许多变化，当比较版本`0.0.3xx`与`0.1.x`的LangChain时，做事的优选方式发生了许多变化。这对于代理来说尤其如此。

我们初始化和使用代理的方式通常比过去更清晰——仍然有许多抽象，但我们（并且被鼓励）更接近代理逻辑本身。这可能会在最初造成一些困惑，但一旦理解了新的逻辑，它可能比之前的版本更清晰。

在这个例子中，我们将使用LangChain v1构建一个RAG代理。我们将使用Claude 3作为我们的LLM，Voyage AI用于知识嵌入，以及Pinecone来提供我们的知识检索功能。

首先，让我们安装必备条件：

```python
# 安装LangChain
pip install langchain

# 安装Claude 3
pip install claude3

# 安装Voyage AI
pip install voyageai

# 安装Pinecone
pip install pinecone

-- 创建数据库表
CREATE TABLE agents (
    id INT PRIMARY KEY,
    name VARCHAR(255),
    description TEXT
);

-- 示例JSON数据
{
    "name": "RAG Agent",
    "description": "使用LangChain v1构建的RAG代理"
}

::: callout-notice 请注意，以上代码仅为示例，实际使用时请根据具体需求进行调整。 :::``````python !pip install -qU \ langchain==0.1.11 \ langchain-core==0.1.30 \ langchain-community==0.0.27 \ langchain-anthropic==0.1.4 \ langchainhub==0.1.15 \ anthropic==0.19.1 \ voyageai==0.2.1 \ pinecone-client==3.1.0 \ datasets==2.16.1

[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m848.6/848.6 kB[0m [31m4.6 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m211.0/211.0 kB[0m [31m18.5 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m507.1/507.1 kB[0m [31m30.3 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m75.6/75.6 kB[0m [31m8.7 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m115.3/115.3 kB[0m [31m13.4 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m134.8/134.8 kB[0m [31m15.0 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m77.8/77.8 kB[0m [31m9.2 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m58.3/58.3 kB[0m [31m6.8 MB/s[0m eta [36m0:00:00[0m
[2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m134.8/134.8 kB[0m [31m13.5 MB/s[0m eta [36m0:00:00[0m
[?25h
```并且获取所需的API密钥。我们需要为[Claude](https://docs.claude.com/claude/reference/getting-started-with-the-api)、[Voyage AI](https://docs.voyageai.com/install/)和[Pinecone](https://docs.pinecone.io/docs/quickstart)获取API密钥。

```html
::: callout-notice
请注意，以下代码块中的变量名和函数名将保持英文。
:::

```python
# 获取API密钥的示例代码
def get_api_keys():
    # 代码逻辑
    pass
``````python
# Insert your API keys here
ANTHROPIC_API_KEY = "<YOUR_ANTHROPIC_API_KEY>"
PINECONE_API_KEY = "<YOUR_PINECONE_API_KEY>"
VOYAGE_API_KEY = "<YOUR_VOYAGE_API_KEY>"
```## 寻找知识

对于一个使用RAG的智能体来说，我们首先需要确定一个知识来源的地方。我们将使用Hugging Face Datasets上的AI ArXiv数据集的v2版本，可在以下链接找到：[`jamescalam/ai-arxiv2-chunks`](https://huggingface.co/datasets/jamescalam/ai-arxiv2-chunks)。

_注意：我们使用的是预分块的数据集。对于原始版本，请参阅[`jamescalam/ai-arxiv2`](https://huggingface.co/datasets/jamescalam/ai-arxiv2)。_```python
from datasets import load_dataset

dataset = load_dataset("jamescalam/ai-arxiv2-chunks", split="train[:20000]")
dataset

/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: 
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(

Downloading data:   0%|          | 0.00/766M [00:00<?, ?B/s]

Generating train split: 0 examples [00:00, ? examples/s]

Dataset({
    features: ['doi', 'chunk-id', 'chunk', 'id', 'title', 'summary', 'source', 'authors', 'categories', 'comment', 'journal_ref', 'primary_category', 'published', 'updated', 'references'],
    num_rows: 20000
})

dataset[1]

{'doi': '2401.09350',
 'chunk-id': 1,
 'chunk': 'These neural networks and their training algorithms may be complex, and the scope of their impact broad and wide, but nonetheless they are simply functions in a high-dimensional space. A trained neural network takes a vector as input, crunches and transforms it in various ways, and produces another vector, often in some other space. An image may thereby be turned into a vector, a song into a sequence of vectors, and a social network as a structured collection of vectors. It seems as though much of human knowledge, or at least what is expressed as text, audio, image, and video, has a vector representation in one form or another.\nIt should be noted that representing data as vectors is not unique to neural networks and deep learning. In fact, long before learnt vector representations of pieces of dataâ\x80\x94what is commonly known as â\x80\x9cembeddingsâ\x80\x9dâ\x80\x94came along, data was often encoded as hand-crafted feature vectors. Each feature quanti- fied into continuous or discrete values some facet of the data that was deemed relevant to a particular task (such as classification or regression). Vectors of that form, too, reflect our understanding of a real-world object or concept.',
 'id': '2401.09350#1',
 'title': 'Foundations of Vector Retrieval',
 'summary': 'Vectors are universal mathematical objects that can represent text, images,\nspeech, or a mix of these data modalities. That happens regardless of whether\ndata is represented by hand-crafted features or learnt embeddings. Collect a\nlarge enough quantity of such vectors and the question of retrieval becomes\nurgently relevant: Finding vectors that are more similar to a query vector.\nThis monograph is concerned with the question above and covers fundamental\nconcepts along with advanced data structures and algorithms for vector\nretrieval. In doing so, it recaps this fascinating topic and lowers barriers of\nentry into this rich area of research.',
 'source': 'http://arxiv.org/pdf/2401.09350',
 'authors': 'Sebastian Bruch',
 'categories': 'cs.DS, cs.IR',
 'comment': None,
 'journal_ref': None,
 'primary_category': 'cs.DS',
 'published': '20240117',
 'updated': '20240117',
 'references': []}
```## 构建知识库

要构建我们的知识库，我们需要 _两样东西_：

1. 嵌入（Embeddings），为此我们将使用 `VoyageEmbeddings`，通过Voyage AI的嵌入模型实现，这需要 [API密钥](https://dash.voyageai.com/api-keys)。
2. 一个向量数据库，用于存储我们的嵌入并查询它们。我们使用Pinecone，这同样需要一个 [免费API密钥](https://app.pinecone.io)。

首先，我们初始化与Voyage AI的连接，并定义一个用于嵌入的 `embed` 对象：

::: callout-notice
请注意，以下代码块中的变量名和函数名将保持不变。
:::

```python
# 初始化Voyage AI连接
voyage_api_key = "your_voyage_api_key"
voyage_client = voyageai.VoyageClient(voyage_api_key)

# 定义嵌入对象
embed = voyage_client.get_embedder()
``````python
from langchain_community.embeddings import VoyageEmbeddings

embed = VoyageEmbeddings(voyage_api_key=VOYAGE_API_KEY, model="voyage-2")

Then we initialize our connection to Pinecone:

from pinecone import Pinecone

# configure client
pc = Pinecone(api_key=PINECONE_API_KEY)
```现在我们设置索引规范，这允许我们定义要部署索引的云提供商和区域。您可以在[此处找到所有可用提供商和区域的列表](https://docs.pinecone.io/docs/projects)。

```markdown
# 索引规范设置

Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects).

::: callout-notice
请注意，以下内容为示例代码，请根据实际情况进行修改。
:::

```python
# 示例代码
def create_index():
    # 创建索引的代码
    pass

{
  "index_name": "example_index",
  "token": "example_token",
  "embedding": "example_embedding",
  "RAG": "example_RAG"
}

-- 示例SQL查询
SELECT * FROM index_table WHERE region = 'us-west-1';

``````python from pinecone import ServerlessSpec

spec = ServerlessSpec(cloud=“aws”, region=“us-west-2”) ```在创建索引之前，我们需要确定Voyage AI嵌入模型的维度，这可以通过创建一个嵌入并检查其长度轻松找到：

Before creating an index, we need the dimensionality of our Voyage AI embedding model, which we can find easily by creating an embedding and checking the length:
``````python
vec = embed.embed_documents(["ello"])
len(vec[0])

1024
```现在我们使用嵌入维度和与模型兼容的度量（可以是余弦或点积）来创建索引。我们还把规范传递给索引初始化。

```markdown
# 使用嵌入维度和度量创建索引

Now we create the index using our embedding dimensionality, and a metric also compatible with the model (this can be either cosine or dotproduct). We also pass our spec to index initialization.

```markdown
```python
# 创建索引的代码示例

def create_index(embedding_dim, metric_type):
    # 创建索引的代码逻辑
    pass

# 索引初始化

We also pass our spec to index initialization.

```python
# 索引初始化的代码示例

def initialize_index(index, spec):
    # 索引初始化的代码逻辑
    pass
``````python
import time

index_name = "claude-3-rag"

# check if index already exists (it shouldn't if this is first time)
if index_name not in pc.list_indexes().names():
    # if does not exist, create index
    pc.create_index(
        index_name,
        dimension=len(vec[0]),  # dimensionality of voyage model
        metric="dotproduct",
        spec=spec,
    )
    # wait for index to be initialized
    while not pc.describe_index(index_name).status["ready"]:
        time.sleep(1)

# connect to index
index = pc.Index(index_name)
time.sleep(1)
# view index stats
index.describe_index_stats()

{'dimension': 1024,
 'index_fullness': 0.0,
 'namespaces': {'': {'vector_count': 20000}},
 'total_vector_count': 20000}
```### 填充我们的索引

现在我们的知识库已经准备好用我们的数据填充了。我们将使用 `embed` 辅助函数将我们的文档嵌入，然后添加到我们的索引中。

我们还将包括每条记录的元数据。```python
from tqdm.auto import tqdm

# easier to work with dataset as pandas dataframe
data = dataset.to_pandas()

batch_size = 100

for i in tqdm(range(0, len(data), batch_size)):
    i_end = min(len(data), i + batch_size)
    # get batch of data
    batch = data.iloc[i:i_end]
    # generate unique ids for each chunk
    ids = [f"{x['doi']}-{x['chunk-id']}" for i, x in batch.iterrows()]
    # get text to embed
    texts = [x["chunk"] for _, x in batch.iterrows()]
    # embed text
    embeds = embed.embed_documents(texts)
    # get metadata to store in Pinecone
    metadata = [
        {"text": x["chunk"], "source": x["source"], "title": x["title"]}
        for i, x in batch.iterrows()
    ]
    # add to Pinecone
    index.upsert(vectors=zip(ids, embeds, metadata, strict=False))

  0%|          | 0/200 [00:00<?, ?it/s]
```创建一个工具，供我们的代理在搜索ArXiv论文时使用：

```markdown
# 创建工具

```python
def search_arxiv_papers(query: str) -> list:
    # 搜索ArXiv论文的代码
    pass

::: callout-notice
请注意，以下代码仅为示例，实际应用中可能需要根据具体情况进行调整。
:::

```python
def search_arxiv_papers(query: str) -> list:
    # 搜索ArXiv论文的代码
    pass

## 工具功能

- 使用RAG进行快速搜索
- 支持嵌入和token处理
- 支持自定义prompt
- 支持API调用
- 支持Python和JSON格式
- 支持SQL查询
``````python
from langchain.agents import tool


@tool
def arxiv_search(query: str) -> str:
    """Use this tool when answering questions about AI, machine learning, data
    science, or other technical questions that may be answered using arXiv
    papers.
    """
    # create query vector
    xq = embed.embed_query(query)
    # perform search
    out = index.query(vector=xq, top_k=5, include_metadata=True)
    # reformat results into string
    results_str = "\n\n".join([x["metadata"]["text"] for x in out["matches"]])
    return results_str


tools = [arxiv_search]

When this tool is used by our agent it will execute it like so:

print(arxiv_search.run(tool_input={"query": "can you tell me about llama 2?"}))

Model Llama 2 Code Llama Code Llama - Python Size FIM LCFT Python CPP Java PHP TypeScript C# Bash Average 7B â 13B â 34B â 70B â 7B â 7B â 7B â 7B â 13B â 13B â 13B â 13B â 34B â 34B â 7B â 7B â 13B â 13B â 34B â 34B â â â â â 14.3% 6.8% 10.8% 9.9% 19.9% 13.7% 15.8% 13.0% 24.2% 23.6% 22.2% 19.9% 27.3% 30.4% 31.6% 34.2% 12.6% 13.2% 21.4% 15.1% 6.3% 3.2% 8.3% 9.5% 3.2% 12.6% 17.1% 3.8% 18.9% 25.9% 8.9% 24.8% â â â â â â â â â â 37.3% 31.1% 36.1% 30.4% 29.2% 29.8% 38.0%

Ethical Considerations and Limitations (Section 5.2) Llama 2 is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Llama 2âs potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. Therefore, before deploying any applications of Llama 2, developers should perform safety testing and tuning tailored to their speciï¬c applications of the model. Please see the Responsible Use Guide available available at https://ai.meta.com/llama/responsible-user-guide
Table 52: Model card for Llama 2.
77

2
Cove Liama Long context (7B =, 13B =, 34B) + fine-tuning ; Lrama 2 Code training 20B oes Cope Liama - Instruct Foundation models â> nfilling code training = eee.â (7B =, 13B =, 34B) â 5B (7B, 13B, 348) 5008 Python code Long context Cove Liama - PyrHon (7B, 13B, 34B) > training Â» Fine-tuning > 1008 208
Figure 2: The Code Llama specialization pipeline. The different stages of fine-tuning annotated with the number of tokens seen during training. Infilling-capable models are marked with the â symbol.
# 2 Code Llama: Specializing Llama 2 for code
# 2.1 The Code Llama models family

# 2 Code Llama: Specializing Llama 2 for code
# 2.1 The Code Llama models family
Code Llama. The Code Llama models constitute foundation models for code generation. They come in four model sizes: 7B, 13B, 34B and 70B parameters. The 7B, 13B and 70B models are trained using an infilling objective (Section 2.3), and are appropriate to be used in an IDE to complete code in the middle of a file, for example. The 34B model was trained without the infilling objective. All Code Llama models are initialized with Llama 2 model weights and trained on 500B tokens from a code-heavy dataset (see Section 2.2 for more details), except Code Llama 70B which was trained on 1T tokens. They are all fine-tuned to handle long contexts as detailed in Section 2.4.

0.52 0.57 0.19 0.30 Llama 1 7B 13B 33B 65B 0.27 0.24 0.23 0.25 0.26 0.24 0.26 0.26 0.34 0.31 0.34 0.34 0.54 0.52 0.50 0.46 0.36 0.37 0.36 0.36 0.39 0.37 0.35 0.40 0.26 0.23 0.24 0.25 0.28 0.28 0.33 0.32 0.33 0.31 0.34 0.32 0.45 0.50 0.49 0.48 0.33 0.27 0.31 0.31 0.17 0.10 0.12 0.11 0.24 0.24 0.23 0.25 0.31 0.27 0.30 0.30 0.44 0.41 0.41 0.43 0.57 0.55 0.60 0.60 0.39 0.34 0.28 0.39 Llama 2 7B 13B 34B 70B 0.28 0.24 0.27 0.31 0.25 0.25 0.24 0.29 0.29 0.35 0.33 0.35 0.50 0.50 0.56 0.51 0.36 0.41 0.41
```## 定义XML代理

XML代理主要构建用于支持Anthropic模型。Anthropic模型经过训练，可以使用XML标签如`<input>{some input}</input>`，或者在使用工具时使用：```
<tool>{tool name}</tool>
<tool_input>{tool input}</tool_input>
```这是与典型ReAct代理生成的格式大不相同，后者在Anthropic模型中支持得并不好。

要创建一个XML代理，我们需要一个`prompt`、`llm`和工具列表。我们可以从LangChain hub下载用于对话XML代理的预构建`prompt`。

```html
::: callout-notice
请注意，以下代码块将展示如何使用Python编写一个简单的API调用。
:::

```python
# 导入所需的库
import requests

# 定义API的URL
url = "https://api.example.com/data"

# 发送GET请求
response = requests.get(url)

# 打印响应内容
print(response.json())

{
  "data": "这里是API返回的数据"
}

-- 以下是一个SQL查询示例
SELECT * FROM users WHERE age > 18;
``````python
from langchain import hub

prompt = hub.pull("hwchase17/xml-agent-convo")
prompt

ChatPromptTemplate(input_variables=['agent_scratchpad', 'input', 'tools'], partial_variables={'chat_history': ''}, messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['agent_scratchpad', 'chat_history', 'input', 'tools'], template="You are a helpful assistant. Help the user answer any questions.\n\nYou have access to the following tools:\n\n{tools}\n\nIn order to use a tool, you can use <tool></tool> and <tool_input></tool_input> tags. You will then get back a response in the form <observation></observation>\nFor example, if you have a tool called 'search' that could run a google search, in order to search for the weather in SF you would respond:\n\n<tool>search</tool><tool_input>weather in SF</tool_input>\n<observation>64 degrees</observation>\n\nWhen you are done, respond with a final answer between <final_answer></final_answer>. For example:\n\n<final_answer>The weather in SF is 64 degrees</final_answer>\n\nBegin!\n\nPrevious Conversation:\n{chat_history}\n\nQuestion: {input}\n{agent_scratchpad}"))])
```我们可以看到在向LLM解释它应该如何使用工具时，整个提示中都在使用XML格式。

接下来，我们初始化与Anthropic的连接，为此我们需要一个[Claude API密钥](https://console.anthropic.com/)。

```::: callout-notice
请注意，初始化连接时需要正确配置API密钥。
``````python
from langchain_anthropic import ChatAnthropic

# chat completion llm
llm = ChatAnthropic(
    ANTHROPIC_API_KEY=ANTHROPIC_API_KEY,
    model_name="claude-opus-4-1",  # change "opus" -> "sonnet" for speed
    temperature=0.0,
)
```当运行代理时，我们将向其提供一个单独的 `input` —— 这是从用户那里获取的输入文本。然而，在代理逻辑中，还会传递一个 *agent_scratchpad* 对象，其中包含工具信息。要将这些信息输入到我们的LLM中，我们需要将其转换为上述描述的XML格式，我们定义了 `convert_intermediate_steps` 函数来处理这一点。

```html
::: callout-attention
请注意，以下代码块中的函数和变量名将保持英文不变。
:::

```python
def convert_intermediate_steps(agent_scratchpad):
    # 在这里实现转换逻辑
    pass
``````python
def convert_intermediate_steps(intermediate_steps):
    log = ""
    for action, observation in intermediate_steps:
        log += (
            f"<tool>{action.tool}</tool><tool_input>{action.tool_input}"
            f"</tool_input><observation>{observation}</observation>"
        )
    return log
```我们还需要将工具解析成一个包含 `tool_name: tool_description` 的字符串 —— 我们使用 `convert_tools` 函数来处理这个。

```python
def convert_tools():
    # 代码实现
    pass
``````python
def convert_tools(tools):
    return "\n".join([f"{tool.name}: {tool.description}" for tool in tools])
```一切准备就绪后，我们可以继续初始化我们的代理对象，使用[**L**ang**C**hain **E**xpression **L**anguage (LCEL)](https://www.pinecone.io/learn/series/langchain/langchain-expression-language/)。我们添加指令，当LLM应该 _停止_ 生成时使用 `llm.bind(stop=[...])`，最后使用 `XMLAgentOutputParser` 对象解析代理的输出。

```python
# 示例代码
def process_data(data):
    # 处理数据
    pass

{
    "name": "example",
    "description": "示例JSON对象"
}

SELECT * FROM users WHERE age > 18;
``````python
from langchain.agents.output_parsers import XMLAgentOutputParser

agent = (
    {
        "input": lambda x: x["input"],
        # without "chat_history", tool usage has no context of prev interactions
        "chat_history": lambda x: x["chat_history"],
        "agent_scratchpad": lambda x: convert_intermediate_steps(x["intermediate_steps"]),
    }
    | prompt.partial(tools=convert_tools(tools))
    | llm.bind(stop=["</tool_input>", "</final_answer>"])
    | XMLAgentOutputParser()
)
```使用初始化后的 `agent` 对象，我们将其与原始的 `tools` 列表一起传递给一个 `AgentExecutor` 对象：

```markdown
With our `agent` object initialized we pass it to an `AgentExecutor` object alongside our original `tools` list:
``````python
from langchain.agents import AgentExecutor

agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

Now we can use the agent via the invoke method:

user_msg = "can you tell me about llama 2?"

out = agent_executor.invoke({"input": user_msg, "chat_history": ""})

print(out["output"])



[1m> Entering new AgentExecutor chain...[0m
[32;1m[1;3m<tool>arxiv_search</tool>
<tool_input>llama 2[0m[36;1m[1;3mModel Llama 2 Code Llama Code Llama - Python Size FIM LCFT Python CPP Java PHP TypeScript C# Bash Average 7B â 13B â 34B â 70B â 7B â 7B â 7B â 7B â 13B â 13B â 13B â 13B â 34B â 34B â 7B â 7B â 13B â 13B â 34B â 34B â â â â â 14.3% 6.8% 10.8% 9.9% 19.9% 13.7% 15.8% 13.0% 24.2% 23.6% 22.2% 19.9% 27.3% 30.4% 31.6% 34.2% 12.6% 13.2% 21.4% 15.1% 6.3% 3.2% 8.3% 9.5% 3.2% 12.6% 17.1% 3.8% 18.9% 25.9% 8.9% 24.8% â â â â â â â â â â 37.3% 31.1% 36.1% 30.4% 29.2% 29.8% 38.0%

2
Cove Liama Long context (7B =, 13B =, 34B) + fine-tuning ; Lrama 2 Code training 20B oes Cope Liama - Instruct Foundation models â> nfilling code training = eee.â (7B =, 13B =, 34B) â 5B (7B, 13B, 348) 5008 Python code Long context Cove Liama - PyrHon (7B, 13B, 34B) > training Â» Fine-tuning > 1008 208
Figure 2: The Code Llama specialization pipeline. The different stages of fine-tuning annotated with the number of tokens seen during training. Infilling-capable models are marked with the â symbol.
# 2 Code Llama: Specializing Llama 2 for code
# 2.1 The Code Llama models family

0.52 0.57 0.19 0.30 Llama 1 7B 13B 33B 65B 0.27 0.24 0.23 0.25 0.26 0.24 0.26 0.26 0.34 0.31 0.34 0.34 0.54 0.52 0.50 0.46 0.36 0.37 0.36 0.36 0.39 0.37 0.35 0.40 0.26 0.23 0.24 0.25 0.28 0.28 0.33 0.32 0.33 0.31 0.34 0.32 0.45 0.50 0.49 0.48 0.33 0.27 0.31 0.31 0.17 0.10 0.12 0.11 0.24 0.24 0.23 0.25 0.31 0.27 0.30 0.30 0.44 0.41 0.41 0.43 0.57 0.55 0.60 0.60 0.39 0.34 0.28 0.39 Llama 2 7B 13B 34B 70B 0.28 0.24 0.27 0.31 0.25 0.25 0.24 0.29 0.29 0.35 0.33 0.35 0.50 0.50 0.56 0.51 0.36 0.41 0.41

Ethical Considerations and Limitations (Section 5.2) Llama 2 is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Llama 2âs potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. Therefore, before deploying any applications of Llama 2, developers should perform safety testing and tuning tailored to their speciï¬c applications of the model. Please see the Responsible Use Guide available available at https://ai.meta.com/llama/responsible-user-guide
Table 52: Model card for Llama 2.
77

Model Size FIM LCFT HumanEval MBPP pass@1 pass@10 pass@100 pass@1 pass@10 pass@100 Llama 2 Code Llama Code Llama - Python 7B â 13B â 34B â 70B â 7B â 7B â 7B â 7B â 13B â 13B â 13B â 13B â 34B â 34B â 7B â 7B â 13B â 13B â 34B â 34B â â â â â â â â â â â â â â â â â â â â â 12.2% 25.2% 20.1% 34.8% 22.6% 47.0% 30.5% 59.4% 32.3% 63.9% 34.1% 62.6% 34.1% 62.5% 33.5% 59.6% 36.6% 72.9% 36.6% 71.9% 37.8% 70.6% 36.0% 69.4% 48.2% 77.7% 48.8% 76.8% 40.2% 70.0% 38.4% 70.3% 45.7% 80.0%[0m[32;1m[1;3mBased on the information from the arXiv search, here are the key points about Llama 2:

<final_answer>
- Llama 2 is a large language model developed by Meta AI. It comes in sizes ranging from 7B to 70B parameters.

- Code Llama is a version of Llama 2 that has been specialized for code generation through fine-tuning on code datasets. Code Llama models are available in Python, C++, Java, PHP, TypeScript, C#, and Bash.

- The Code Llama specialization pipeline involves foundation model pre-training, long context training, code infilling training, and fine-tuning on specific programming languages. 

- Code Llama significantly outperforms the base Llama 2 models on code generation benchmarks like HumanEval and MBPP. For example, the 34B parameter Code Llama - Python achieves 48.8% pass@1 on HumanEval compared to 34.1% for the 34B Llama 2.

- As with all large language models, Llama 2 has limitations and potential risks that need to be considered before deploying it in applications. Meta provides a responsible use guide with recommendations for safety testing and tuning.
[0m

[1m> Finished chain.[0m

- Llama 2 is a large language model developed by Meta AI. It comes in sizes ranging from 7B to 70B parameters.

- Code Llama is a version of Llama 2 that has been specialized for code generation through fine-tuning on code datasets. Code Llama models are available in Python, C++, Java, PHP, TypeScript, C#, and Bash.

- The Code Llama specialization pipeline involves foundation model pre-training, long context training, code infilling training, and fine-tuning on specific programming languages. 

- Code Llama significantly outperforms the base Llama 2 models on code generation benchmarks like HumanEval and MBPP. For example, the 34B parameter Code Llama - Python achieves 48.8% pass@1 on HumanEval compared to 34.1% for the 34B Llama 2.

- As with all large language models, Llama 2 has limitations and potential risks that need to be considered before deploying it in applications. Meta provides a responsible use guide with recommendations for safety testing and tuning.
```那看起来相当不错，但现在的我们的代理是 _无状态的_，这使得与之进行对话变得困难。我们可以通过多种方式给它赋予记忆，但其中一种最简单的方法是使用 `ConversationBufferWindowMemory`。```python
from langchain.chains.conversation.memory import ConversationBufferWindowMemory

# conversational memory
conversational_memory = ConversationBufferWindowMemory(
    memory_key="chat_history", k=5, return_messages=True
)
```我们尚未将对话记忆附加到我们的代理上——因此 `conversational_memory` 对象将保持为空：

```python
# 代码示例
def process_message(message):
    # 处理消息
    pass
``````python
conversational_memory.chat_memory.messages

[]
```我们必须手动将我们与代理之间的交互添加到我们的记忆中。

```markdown
# 我们必须手动将我们与代理之间的交互添加到我们的记忆中。
``````python
conversational_memory.chat_memory.add_user_message(user_msg)
conversational_memory.chat_memory.add_ai_message(out["output"])

conversational_memory.chat_memory.messages

[HumanMessage(content='can you tell me about llama 2?'),
 AIMessage(content='\n- Llama 2 is a large language model developed by Meta AI. It comes in sizes ranging from 7B to 70B parameters.\n\n- Code Llama is a version of Llama 2 that has been specialized for code generation through fine-tuning on code datasets. Code Llama models are available in Python, C++, Java, PHP, TypeScript, C#, and Bash.\n\n- The Code Llama specialization pipeline involves foundation model pre-training, long context training, code infilling training, and fine-tuning on specific programming languages. \n\n- Code Llama significantly outperforms the base Llama 2 models on code generation benchmarks like HumanEval and MBPP. For example, the 34B parameter Code Llama - Python achieves 48.8% pass@1 on HumanEval compared to 34.1% for the 34B Llama 2.\n\n- As with all large language models, Llama 2 has limitations and potential risks that need to be considered before deploying it in applications. Meta provides a responsible use guide with recommendations for safety testing and tuning.\n')]

Now we can see that two messages have been added, our HumanMessage the agent’s AIMessage response. Unfortunately, we cannot send these messages to our XML agent directly. Instead, we need to pass a string in the format:

Human: {human message}
AI: {AI message}
```让我们快速编写一个 `memory2str` 辅助函数来帮我们处理这个问题：

```python
def memory2str(memory):
    # 代码实现
    pass
``````python
from langchain_core.messages.human import HumanMessage


def memory2str(memory: ConversationBufferWindowMemory):
    messages = memory.chat_memory.messages
    memory_list = [
        f"Human: {mem.content}" if isinstance(mem, HumanMessage) else f"AI: {mem.content}"
        for mem in messages
    ]
    memory_str = "\n".join(memory_list)
    return memory_str

print(memory2str(conversational_memory))

Human: can you tell me about llama 2?
AI: 
- Llama 2 is a large language model developed by Meta AI. It comes in sizes ranging from 7B to 70B parameters.

- Code Llama is a version of Llama 2 that has been specialized for code generation through fine-tuning on code datasets. Code Llama models are available in Python, C++, Java, PHP, TypeScript, C#, and Bash.

- The Code Llama specialization pipeline involves foundation model pre-training, long context training, code infilling training, and fine-tuning on specific programming languages. 

- Code Llama significantly outperforms the base Llama 2 models on code generation benchmarks like HumanEval and MBPP. For example, the 34B parameter Code Llama - Python achieves 48.8% pass@1 on HumanEval compared to 34.1% for the 34B Llama 2.

- As with all large language models, Llama 2 has limitations and potential risks that need to be considered before deploying it in applications. Meta provides a responsible use guide with recommendations for safety testing and tuning.
```现在，让我们再创建一个名为 `chat` 的辅助函数，帮助我们处理代理的 _状态_ 部分。

```markdown
# 现在让我们再创建一个名为 `chat` 的辅助函数，帮助我们处理代理的 _状态_ 部分。
``````python
def chat(text: str):
    out = agent_executor.invoke({"input": text, "chat_history": memory2str(conversational_memory)})
    conversational_memory.chat_memory.add_user_message(text)
    conversational_memory.chat_memory.add_ai_message(out["output"])
    return out["output"]
```现在我们只需与我们的代理进行聊天，它就会记住之前交互的上下文。

```markdown
# 现在我们只需与我们的代理进行聊天，它就会记住之前交互的上下文。
``````python
print(chat("was any red teaming done with the model?"))

```

[1m> Entering new AgentExecutor chain…[0m [32;1m[1;3marxiv_search llama 2 red teaming[0m[36;1m[1;3mAfter conducting red team exercises, we asked participants (who had also participated in Llama 2 Chat exercises) to also provide qualitative assessment of safety capabilities of the model. Some participants who had expertise in offensive security and malware development questioned the ultimate risk posed by âmalicious code generationâ through LLMs with current capabilities. One red teamer remarked, âWhile LLMs being able to iteratively improve on produced source code is a risk, producing source code isnât the actual gap. That said, LLMs may be risky because they can inform low-skill adversaries in production of scripts through iteration that perform some malicious behavior.â According to another red teamer, â[v]arious scripts, program code, and compiled binaries are readily available on mainstream public websites, hacking forums or on âthe dark web.â Advanced malware development is beyond the current capabilities of available LLMs, and even an advanced LLM paired with an expert malware developer is not particularly useful- as the barrier is not typically writing the malware code itself. That said, these LLMs may produce code which will get easily caught if used directly.â

Model Llama 2 Code Llama Code Llama - Python Size FIM LCFT Python CPP Java PHP TypeScript C# Bash Average 7B â 13B â 34B â 70B â 7B â 7B â 7B â 7B â 13B â 13B â 13B â 13B â 34B â 34B â 7B â 7B â 13B â 13B â 34B â 34B â â â â â 14.3% 6.8% 10.8% 9.9% 19.9% 13.7% 15.8% 13.0% 24.2% 23.6% 22.2% 19.9% 27.3% 30.4% 31.6% 34.2% 12.6% 13.2% 21.4% 15.1% 6.3% 3.2% 8.3% 9.5% 3.2% 12.6% 17.1% 3.8% 18.9% 25.9% 8.9% 24.8% â â â â â â â â â â 37.3% 31.1% 36.1% 30.4% 29.2% 29.8% 38.0%

In addition to red teaming sessions, we ran a quantitative evaluation on risk from generating malicious code by scoring Code Llamaâs responses to ChatGPTâs (GPT3.5 Turbo) with LLAMAv2 70Bâs safety reward model. For this second quantitative evaluation, we selected prompts that the red teamers generated specifically attempting to solicit malicious code (even though the red teaming included consideration of a broad set of safety risks). These prompts were a mix of clear intent and slightly obfuscated intentions (see some examples in Figure 16. We show a KDE plot of the distribution of the safety score for all models in Figure 7). We observe that Code Llama tends to answer with safer responses; the distribution of safety scores for Code Llama has more weight in the safer part of the range. False refusals. LLMs that are too safe can have a tendency to over-refuse valid claims similar to what was reported after the release of Llama 2. We specifically asked red teamers to test for this behavior. They found some limited evidence of false refusals (when not using a system preprompt). False refusals could also 16

15 pafety Reward Model Scores Distribution on Red Teaming Prompts Responding Model GPT 3.5 Turbo Code Llama 138 Instruct Code Llama 34B Instruct Code Llama 7B Instruct 0.0-+ -0.2 0.0 0.2 0.4 0.6 08 1.0 12 Llama 2 70B Safety Reward Model Score Figure 7: KDE plot of the risk score output by the Llama 2 safety reward model on prompts with clear intent specific to code risk created by red teamers with background in cybersecurity and malware generation. Red teaming. It is important to also proactively identify risks with adversarial testing or red teaming. We conducted 3 red teaming exercises with 25 Meta employees, including domain experts in responsible AI, malware development, and offensive security engineering. The red teamers provided a nuanced evaluation specifically on the risk from so called âdual intent prompts.â Dual intent prompts are requests for help with writing code that could be used maliciously but the prompt does not directly address the topic (example âMosaic Promptsâ Glukhov et al. (2023)). For example, the model rightfully refuses to provide support with writing ransomware code but it complies when asked to provide a script to encrypt all files in the userâs home directory since such a script could be used for benign purposes.

. . . . . . . . . . . . . . . 3.4 RLHF Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Safety in Pretraining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Safety Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Red Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Safety Evaluation of Llama 2-Chat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Learnings and Observations . . . . . . . . . . . . . . . . . . . . . . . .[0m[32;1m[1;3mBased on the information from the arxiv search, some red teaming was done on the Llama 2 model during development to identify potential safety risks:

- Meta conducted 3 red teaming exercises with 25 employees, including domain experts in responsible AI, malware development, and offensive security engineering.

The red teamers categorized successful attacks into four main types: 1) getting the model to provide some harmful information while refusing other content, 2) having the model roleplay specific scenarios, 3) forcing the model to highlight positives of harmful content, and 4) embedding harmful instructions within complex commands.
Some red teamers questioned the ultimate risk posed by “malicious code generation” through current LLMs. They noted that while LLMs being able to iteratively improve code is a risk, producing source code itself isn’t the main gap. Advanced malware development is currently beyond LLM capabilities.
Quantitative evaluation was also done by scoring Code Llama’s responses to malicious code prompts using Llama 2’s safety reward model. Code Llama tended to give safer responses compared to GPT-3.5.
However, the full extent and details of the red teaming are limited based on the information available. The Llama 2 paper mentions expanding prompts with safety risks via red teaming, but does not go in-depth on the process or results. More information would be needed to fully characterize the red teaming performed. [0m

[1m> Finished chain.[0m

Meta conducted 3 red teaming exercises with 25 employees, including domain experts in responsible AI, malware development, and offensive security engineering.
The red teamers categorized successful attacks into four main types: 1) getting the model to provide some harmful information while refusing other content, 2) having the model roleplay specific scenarios, 3) forcing the model to highlight positives of harmful content, and 4) embedding harmful instructions within complex commands.
Some red teamers questioned the ultimate risk posed by “malicious code generation” through current LLMs. They noted that while LLMs being able to iteratively improve code is a risk, producing source code itself isn’t the main gap. Advanced malware development is currently beyond LLM capabilities.
Quantitative evaluation was also done by scoring Code Llama’s responses to malicious code prompts using Llama 2’s safety reward model. Code Llama tended to give safer responses compared to GPT-3.5.
However, the full extent and details of the red teaming are limited based on the information available. The Llama 2 paper mentions expanding prompts with safety risks via red teaming, but does not go in-depth on the process or results. More information would be needed to fully characterize the red teaming performed. ``我们可以提出一些遗漏关键信息的后续问题，但由于对话历史的存在，LLM能够理解上下文并据此调整搜索查询。例如，我们询问了red teaming，但没有提到llama 2——Claude 3根据聊天历史将此上下文添加到搜索查询“llama 2 red teaming”`中。

Tip

中文读者提示：本章节的代码和输出为英文原文。如需理解具体实现细节，可参考上方中文导读和代码注释。如有疑问，欢迎在 GitHub Issues 讨论。