教程：多跳检索¶
让我们快速浏览一个构建具有多个子模块的 dspy.Module 的示例。我们将为此任务进行多跳搜索。
通过 pip install -U dspy 安装最新版本的 DSPy，并按步骤操作。
推荐：设置 MLflow 跟踪，以了解内部运行情况。
MLflow DSPy 集成¶
MLflow 是一个 LLMOps 工具，与 DSPy 原生集成，提供可解释性和实验跟踪功能。在本教程中，您可以使用 MLflow 将提示和优化进度可视化为跟踪，以更好地理解 DSPy 的行为。您可以通过以下四个步骤轻松设置 MLflow。
- 安装 MLflow
%pip install mlflow>=2.20
- 在单独的终端中启动 MLflow UI
mlflow ui --port 5000
- 将 notebook 连接到 MLflow
import mlflow
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("DSPy")
- 启用跟踪。
mlflow.dspy.autolog()
要了解更多关于集成的信息，请访问MLflow DSPy 文档。
在本教程中，我们将使用一个小型本地语言模型，Meta 的 Llama-3.1-8B-Instruct，它有 80 亿个参数。
您可能可以使用 Ollama 在您的笔记本电脑上托管 8B 模型，或使用 SGLang 在您的 GPU 服务器上托管，或者通过 Databricks 或 Together 等提供商来托管。
在下面的代码片段中，我们将把这个小型模型配置为我们的主语言模型。我们还将设置一个更大的语言模型，即 GPT-4o，作为教师模型，我们将调用它很少的次数来帮助训练小型语言模型。这在技术上不是必需的；小型模型通常可以在 DSPy 中自行学习此类任务。但使用更大的教师模型会让我们更安心，因为初始系统或优化器配置的影响不会那么大。
输入 [1]
已复制！
import dspy
lm = dspy.LM('<your_provider>/Llama-3.1-8B-Instruct', max_tokens=3000)
gpt4o = dspy.LM('openai/gpt-4o', max_tokens=3000)
dspy.configure(lm=lm)
import dspy lm = dspy.LM('/Llama-3.1-8B-Instruct', max_tokens=3000) gpt4o = dspy.LM('openai/gpt-4o', max_tokens=3000) dspy.configure(lm=lm)
安装依赖项并下载数据¶
为了进行检索，我们将使用很棒的 BM25S 库，因为它非常轻量级。您可以根据需要替换此组件。
> pip install -U bm25s PyStemmer "jax[cpu]"
接下来，我们将下载截至 2017 年的所有 5,000,000 个维基百科页面的摘要（即第一段）快照。我们将把它用作我们的检索语料库。
压缩后大小为 500MB，因此下载和解压缩可能需要 2-3 分钟。
from dspy.utils import download
download("https://hugging-face.cn/dspy/cache/resolve/main/wiki.abstracts.2017.tar.gz")
!tar -xzvf wiki.abstracts.2017.tar.gz
现在加载语料库。
输入 [2]
已复制！
import ujson
corpus = []
with open("wiki.abstracts.2017.jsonl") as f:
for line in f:
line = ujson.loads(line)
corpus.append(f"{line['title']} | {' '.join(line['text'])}")
len(corpus)
import ujson corpus = [] with open("wiki.abstracts.2017.jsonl") as f: for line in f: line = ujson.loads(line) corpus.append(f"{line['title']} | {' '.join(line['text'])}") len(corpus)
输出[2]
5233330
然后为 BM25 检索建立索引！这需要 2-3 分钟。
输入 [ ]
已复制！
import bm25s
import Stemmer
stemmer = Stemmer.Stemmer("english")
corpus_tokens = bm25s.tokenize(corpus, stopwords="en", stemmer=stemmer)
retriever = bm25s.BM25(k1=0.9, b=0.4)
retriever.index(corpus_tokens)
import bm25s import Stemmer stemmer = Stemmer.Stemmer("english") corpus_tokens = bm25s.tokenize(corpus, stopwords="en", stemmer=stemmer, show_progress=False) retriever = bm25s.BM25(k1=0.9, b=0.4) retriever.index(corpus_tokens)
加载 HoVer 数据集。¶
为我们的任务加载一个数据集。我们将从 HoVer 多跳任务中加载示例，其中输入是一个（非常！）复杂的断言，我们寻求的输出是事实核查该断言所需的维基百科页面集。
输入 [4]
已复制！
import random
from dspy.datasets import DataLoader
kwargs = dict(fields=("claim", "supporting_facts", "hpqa_id", "num_hops"), input_keys=("claim",))
hover = DataLoader().from_huggingface(dataset_name="hover-nlp/hover", split="train", trust_remote_code=True, **kwargs)
hpqa_ids = set()
hover = [
dspy.Example(claim=x.claim, titles=list(set([y["key"] for y in x.supporting_facts]))).with_inputs("claim")
for x in hover
if x["num_hops"] == 3 and x["hpqa_id"] not in hpqa_ids and not hpqa_ids.add(x["hpqa_id"])
]
random.Random(0).shuffle(hover)
trainset, devset, testset = hover[:200], hover[200:500], hover[650:]
import random from dspy.datasets import DataLoader kwargs = dict(fields=("claim", "supporting_facts", "hpqa_id", "num_hops"), input_keys=("claim",)) hover = DataLoader().from_huggingface(dataset_name="hover-nlp/hover", split="train", trust_remote_code=True, **kwargs) hpqa_ids = set() hover = [ dspy.Example(claim=x.claim, titles=list(set([y["key"] for y in x.supporting_facts]))).with_inputs("claim") for x in hover if x["num_hops"] == 3 and x["hpqa_id"] not in hpqa_ids and not hpqa_ids.add(x["hpqa_id"]) ] random.Random(0).shuffle(hover) trainset, devset, testset = hover[:200], hover[200:500], hover[650:]
查看此任务的一个示例
输入 [5]
已复制！
example = trainset[0]
print("Claim:", example.claim)
print("Pages that must be retrieved:", example.titles)
example = trainset[0] print("Claim:", example.claim) print("Pages that must be retrieved:", example.titles)
Claim: This director is known for his work on Miss Potter. The Academy of Motion Picture Arts and Sciences presents the award in which he was nominated for his work in "Babe". Pages that must be retrieved: ['Miss Potter', 'Chris Noonan', 'Academy Award for Best Director']
现在，定义一个在维基百科中进行搜索的函数。这将使用我们的 BM25 索引。
输入 [6]
已复制！
def search(query: str, k: int) -> list[str]:
tokens = bm25s.tokenize(query, stopwords="en", stemmer=stemmer, show_progress=False)
results, scores = retriever.retrieve(tokens, k=k, n_threads=1, show_progress=False)
run = {corpus[doc]: float(score) for doc, score in zip(results[0], scores[0])}
return run
def search(query: str, k: int) -> list[str]: tokens = bm25s.tokenize(query, stopwords="en", stemmer=stemmer, show_progress=False) results, scores = retriever.retrieve(tokens, k=k, n_threads=1, show_progress=False) run = {corpus[doc]: float(score) for doc, score in zip(results[0], scores[0])} return run
现在，在 DSPy 中定义多跳程序。它将非常简单：接受一个 claim 并生成一个列表 titles: list[str]。
它将通过两个子模块来实现：generate_query 和 append_notes。
输入 [7]
已复制！
class Hop(dspy.Module):
def __init__(self, num_docs=10, num_hops=4):
self.num_docs, self.num_hops = num_docs, num_hops
self.generate_query = dspy.ChainOfThought('claim, notes -> query')
self.append_notes = dspy.ChainOfThought('claim, notes, context -> new_notes: list[str], titles: list[str]')
def forward(self, claim: str) -> list[str]:
notes = []
titles = []
for _ in range(self.num_hops):
query = self.generate_query(claim=claim, notes=notes).query
context = search(query, k=self.num_docs)
prediction = self.append_notes(claim=claim, notes=notes, context=context)
notes.extend(prediction.new_notes)
titles.extend(prediction.titles)
return dspy.Prediction(notes=notes, titles=list(set(titles)))
class Hop(dspy.Module): def __init__(self, num_docs=10, num_hops=4): self.num_docs, self.num_hops = num_docs, num_hops self.generate_query = dspy.ChainOfThought('claim, notes -> query') self.append_notes = dspy.ChainOfThought('claim, notes, context -> new_notes: list[str], titles: list[str]') def forward(self, claim: str) -> list[str]: notes = [] titles = [] for _ in range(self.num_hops): query = self.generate_query(claim=claim, notes=notes).query context = search(query, k=self.num_docs) prediction = self.append_notes(claim=claim, notes=notes, context=context) notes.extend(prediction.new_notes) titles.extend(prediction.titles) return dspy.Prediction(notes=notes, titles=list(set(titles)))
太好了。现在设置一个评估指标，top5_recall。
它将返回程序返回的前 5 个标题中检索到的黄金页面（始终为 3 个）的比例。
输入 [8]
已复制！
def top5_recall(example, pred, trace=None):
gold_titles = example.titles
recall = sum(x in pred.titles[:5] for x in gold_titles) / len(gold_titles)
# If we're "bootstrapping" for optimization, return True if and only if the recall is perfect.
if trace is not None:
return recall >= 1.0
# If we're just doing inference, just measure the recall.
return recall
evaluate = dspy.Evaluate(devset=devset, metric=top5_recall, num_threads=16, display_progress=True, display_table=5)
def top5_recall(example, pred, trace=None): gold_titles = example.titles recall = sum(x in pred.titles[:5] for x in gold_titles) / len(gold_titles) # 如果我们正在进行优化“自举”，则仅在召回率完美时返回 True。 if trace is not None: return recall >= 1.0 # 如果我们只是进行推断，则只测量召回率。 return recall evaluate = dspy.Evaluate(devset=devset, metric=top5_recall, num_threads=16, display_progress=True, display_table=5)
评估我们的现成程序！
输入 [9]
已复制！
evaluate(Hop())
evaluate(Hop())
Average Metric: 27.67 / 98 (28.2%): 32%|███▏ | 97/300 [00:02<00:04, 49.34it/s]
2024/12/25 12:18:00 ERROR dspy.utils.parallelizer: Error processing item Example({'claim': "All That is the show that the co-creator with the host of Vibe and Wild 'N Out had a debut on.", 'titles': ['Chris Spencer (actor)', 'Nick Cannon', 'Vibe (talk show)']}) (input_keys={'claim'}): Expected dict_keys(['reasoning', 'new_notes', 'titles']) but got dict_keys(['reasoning', 'new_notes']). Set `provide_traceback=True` to see the stack trace.
Average Metric: 59.33 / 186 (31.9%): 62%|██████▏ | 186/300 [00:03<00:02, 51.84it/s]
2024/12/25 12:18:02 ERROR dspy.utils.parallelizer: Error processing item Example({'claim': 'The song, which Billie Anthony is best known for her Top 10 hit version, topped the UK chart in 1981 in a recording by a platinum-selling British rock and roll singer whose recording and performing career began in the late 1960s.', 'titles': ["Shakin' Stevens", 'This Ole House', 'Billie Anthony']}) (input_keys={'claim'}): Expected dict_keys(['reasoning', 'new_notes', 'titles']) but got dict_keys(['reasoning']). Set `provide_traceback=True` to see the stack trace.
Average Metric: 94.00 / 298 (31.5%): 100%|██████████| 300/300 [00:06<00:00, 48.56it/s]
2024/12/25 12:18:04 INFO dspy.evaluate.evaluate: Average Metric: 93.99999999999993 / 300 (31.3%)
| 断言 | 示例标题 | 笔记 | 预测标题 | top5_recall | 标题 | |
|---|---|---|---|---|---|---|
| 0 | 耐克足球队有一名球员曾代言耐克足球鞋... | [Nike Hypervenom, Nike Total 90, Marcus Rashford] | ['Nike Total 90 已被 Nike Hypervenom 取代。', 'T...] | ['Nike Mercurial Vapor | Mercurial Vapor 是一款足球鞋，由...'...] | ✔️ [0.333] | NaN |
| 1 | Bill Boyd 是经营该电器公司的董事长... | [Suncoast Hotel and Casino, Boyd Gaming, Thomas Eje] | ['未提及 Bill Boyd 是电器公司的董事长...'...] | [Suncoast Casino, Thomas Eje, Boyd Gaming Corporation, Bill Boyd, ...] | ✔️ [0.333] | NaN |
| 2 | 韩国总统出生于 1953 年 1 月 24 日。该团体... | [Presidential Council on Nation Branding, Korea, Moon Jae-in, Euh ...] | ['韩国总统可能出生在 1945 年之前', 'Euh ...'] | ['李哲承', '韩国总统列表', '寿命...'...] | NaN | |
| 3 | 电影 Khan Kluay 在 2009 年电影上映前 2 个月上映... | [了不起的狐狸爸爸 (电影), Jason Schwartzman, Khan Kluay] | ['电影 Khan Kluay 于 2006 年上映。', '2009 年的电影...'...] | [Khan Kluay, 穿越大吉岭] | ✔️ [0.333] | NaN |
| 4 | 电影《海底总动员 2：多莉去哪儿》的导演曾联合执导电影《虫虫危机》。 | [Andrew Stanton, 海底总动员 2：多莉去哪儿, 虫虫危机] | ['《海底总动员 2：多莉去哪儿》的导演是 Andrew Stanton 和 Angus MacLane...'...] | [海底总动员 2：多莉去哪儿, 虫虫危机] | ✔️ [0.667] | NaN |
... 还有 295 行未显示 ...
输出[9]
31.33
在 MLflow 实验中跟踪评估结果
为了跟踪和可视化随时间变化的评估结果，您可以将结果记录在 MLflow 实验中。
import mlflow
with mlflow.start_run(run_name="hop_evaluation"):
evaluate = dspy.Evaluate(
devset=devset,
metric=top5_recall,
num_threads=16,
display_progress=True,
# To record the outputs and detailed scores to MLflow
return_all_scores=True,
return_outputs=True,
)
# Evaluate the program as usual
aggregated_score, outputs, all_scores = evaluate(Hop())
# Log the aggregated score
mlflow.log_metric("top5_recall", aggregated_score)
# Log the detailed evaluation results as a table
mlflow.log_table(
{
"Claim": [example.claim for example in eval_set],
"Expected Titles": [example.titles for example in eval_set],
"Predicted Titles": outputs,
"Top 5 Recall": all_scores,
},
artifact_file="eval_results.json",
)
要了解更多关于集成的信息，请访问MLflow DSPy 文档。
现在，联合优化 Hop() 程序内的两个提示，以最大化程序的召回率。这可能需要大约 35 分钟，并调用约 5 美元价值的 GPT-4o 来优化 Llama-3.1-8B。
输入 [10]
已复制！
models = dict(prompt_model=gpt4o, teacher_settings=dict(lm=gpt4o))
tp = dspy.MIPROv2(metric=top5_recall, auto="medium", num_threads=16, **models)
kwargs = dict(minibatch_size=40, minibatch_full_eval_steps=4, requires_permission_to_run=False)
optimized = tp.compile(Hop(), trainset=trainset, max_bootstrapped_demos=4, max_labeled_demos=4, **kwargs)
models = dict(prompt_model=gpt4o, teacher_settings=dict(lm=gpt4o)) tp = dspy.MIPROv2(metric=top5_recall, auto="medium", num_threads=16, **models) kwargs = dict(minibatch_size=40, minibatch_full_eval_steps=4, requires_permission_to_run=False) optimized = tp.compile(Hop(), trainset=trainset, max_bootstrapped_demos=4, max_labeled_demos=4, **kwargs)
现在，优化后再次评估。
输入 [11]
已复制！
evaluate(optimized)
evaluate(optimized)
Average Metric: 38.67 / 64 (60.4%): 21%|██ | 63/300 [00:01<00:06, 38.13it/s]
2024/12/25 12:18:09 ERROR dspy.utils.parallelizer: Error processing item Example({'claim': 'Eliot Hyman co-founded Seven Arts Productions in 1957. His co-founder produced the American-American black comedy-drama film directed by Stanley Kubrick.', 'titles': ['Ray Stark', 'Seven Arts Productions', 'Lolita (1962 film)']}) (input_keys={'claim'}): Expected dict_keys(['reasoning', 'query']) but got dict_keys(['reasoning']). Set `provide_traceback=True` to see the stack trace.
Average Metric: 177.33 / 299 (59.3%): 100%|██████████| 300/300 [00:08<00:00, 36.01it/s]
2024/12/25 12:18:16 INFO dspy.evaluate.evaluate: Average Metric: 177.33333333333334 / 300 (59.1%)
| 断言 | 示例标题 | 笔记 | 预测标题 | top5_recall | 标题 | |
|---|---|---|---|---|---|---|
| 0 | 耐克足球队有一名球员曾代言耐克足球鞋... | [Nike Hypervenom, Nike Total 90, Marcus Rashford] | [] | [Nike Hypervenom, Nike Total 90, Kylian Mbappé, Marcus Rashford] | ✔️ [1.000] | NaN |
| 1 | Bill Boyd 是经营该电器公司的董事长... | [Suncoast Hotel and Casino, Boyd Gaming, Thomas Eje] | [] | [Bill Boyd, Suncoast Casino, Las Vegas, Thomas Eje] | ✔️ [0.333] | NaN |
| 2 | 韩国总统出生于 1953 年 1 月 24 日。该团体... | [Presidential Council on Nation Branding, Korea, Moon Jae-in, Euh ...] | ['Euh Yoon-Dae 是一位韩国教授、金融家和顾问...'...] | [Euh Yoon-Dae, KB Financial Group, Chang Dae-hwan, Maeil Business ...] | NaN | |
| 3 | 电影 Khan Kluay 在 2009 年电影上映前 2 个月上映... | [了不起的狐狸爸爸 (电影), Jason Schwartzman, Khan Kluay] | ["Jason Schwartzman 在 2009 年的电影中与 Wes Anderson 合作...'"] | [Wes Anderson, 了不起的狐狸爸爸, Khan Kluay 2, Jason Schwartzman,...] | ✔️ [0.667] | NaN |
| 4 | 电影《海底总动员 2：多莉去哪儿》的导演曾联合执导电影《虫虫危机》。 | [Andrew Stanton, 海底总动员 2：多莉去哪儿, 虫虫危机] | ["Andrew Stanton 联合执导了《虫虫危机》", "John Lasseter 执导了...'"] | [John Lasseter, Andrew Stanton, 海底总动员 2：多莉去哪儿, 虫虫危机] | ✔️ [1.000] | NaN |
... 还有 295 行未显示 ...
输出[11]
59.11
太棒了。看起来系统的召回率从约 30% 大幅提高到接近 60%。这是一个相当直接的方法，但 DSPy 提供了许多工具供您在此基础上继续迭代。
接下来，检查优化的提示，了解它学到了什么。我们将运行一次查询，然后检查最后两个提示，这将向我们展示 Hop() 程序内后续迭代中两个子模块使用的提示。（另外，如果您按照上述说明启用了 MLflow 跟踪，您可以在丰富的树状视图中查看 agent 执行的所有步骤，包括 LLM 调用、提示和工具执行。）
输入 [13]
已复制！
optimized(claim="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles
optimized(claim="为披头士乐队编写的 20 世纪 60 年代未制作的剧本《Up Against It》的作者和 Bernard-Marie Koltès 都是剧作家。").titles
输出[13]
['Up Against It', 'Bernard-Marie Koltès', 'The Beatles', 'Joe Orton']
输入 [15]
已复制！
dspy.inspect_history(n=2)
dspy.inspect_history(n=2)
[2024-12-25T12:18:16.177899] System message: Your input fields are: 1. `claim` (str) 2. `notes` (str) Your output fields are: 1. `reasoning` (str) 2. `query` (str) All interactions will be structured in the following way, with the appropriate values filled in. [[ ## claim ## ]] {claim} [[ ## notes ## ]] {notes} [[ ## reasoning ## ]] {reasoning} [[ ## query ## ]] {query} [[ ## completed ## ]] In adhering to this structure, your objective is: Given a claim and a set of notes, generate a query that can be used to gather additional evidence or context to support or refute the claim. Think step by step to ensure the query is specific and relevant to the information provided in the notes. User message: [[ ## claim ## ]] Danyang, Jiangusu and this city are both cities in China. This city was the birthplace of Chen Xiuke. [[ ## notes ## ]] [1] «Chen Xiuke was born in Dongfang, Hainan.» [2] «Danyang is a city in Jiangsu province, China.» [3] «Chen Xiuke was born in Dongfang, Hainan.» [4] «Danyang is a city in Jiangsu province, China.» [5] «Dongfang is a city in Hainan province, China.» [6] «Chen Xiuke was born in Dongfang, Hainan.» [7] «Danyang is a city in Jiangsu province, China.» [8] «Dongfang is a city in Hainan province, China.» [9] «Chen Xiuke was born in Dongfang, Hainan.» [10] «Danyang is a city in Jiangsu province, China.» [11] «Dongfang is a city in Hainan province, China.» Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## query ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`. Assistant message: [[ ## reasoning ## ]] The claim states that Danyang, Jiangsu, and another city are both cities in China, and that this other city was the birthplace of Chen Xiuke. According to the notes, Chen Xiuke was born in Dongfang, Hainan. Therefore, the city referred to in the claim as "this city" is Dongfang, which is indeed a city in Hainan province, China. The claim is consistent with the information provided in the notes. [[ ## query ## ]] What is the birthplace of Chen Xiuke? [[ ## completed ## ]] User message: [[ ## claim ## ]] The star of Nothing to Report and Gary Barlow have a profession in common. [[ ## notes ## ]] [1] «Chris Jericho is a musician and professional wrestler.» [2] «Gary Barlow is a singer and songwriter, known for being a member of Take That.» [3] «Chris Jericho is a musician and professional wrestler.» [4] «Gary Barlow is a singer and songwriter, known for being a member of Take That.» [5] «Chris Jericho stars in 'Nothing to Report'.» [6] «Chris Jericho is a musician and professional wrestler.» [7] «Gary Barlow is a singer and songwriter, known for being a member of Take That.» [8] «Chris Jericho stars in 'Nothing to Report'.» [9] «Chris Jericho is a musician and professional wrestler.» [10] «Gary Barlow is a singer and songwriter, known for being a member of Take That.» [11] «Chris Jericho stars in 'Nothing to Report'.» Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## query ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`. Assistant message: [[ ## reasoning ## ]] The claim suggests that the star of "Nothing to Report" and Gary Barlow share a profession. From the notes, we know that Chris Jericho stars in "Nothing to Report" and is a musician and professional wrestler. Gary Barlow is a singer and songwriter. The common profession between Chris Jericho and Gary Barlow is that they are both musicians. [[ ## query ## ]] What is the profession that Chris Jericho and Gary Barlow have in common? [[ ## completed ## ]] User message: [[ ## claim ## ]] The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights. [[ ## notes ## ]] N/A Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## query ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`. Response: [[ ## reasoning ## ]] The claim states that the author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights. However, there is no information in the notes to support or refute this claim. The notes are empty, which means we cannot verify the author of the script or Bernard-Marie Koltès' profession. [[ ## query ## ]] Who is the author of the 1960s unproduced script written for The Beatles, Up Against It? [[ ## completed ## ]] [2024-12-25T12:18:16.191896] System message: Your input fields are: 1. `claim` (str) 2. `notes` (str) 3. `context` (str) Your output fields are: 1. `reasoning` (str) 2. `new_notes` (list[str]) 3. `titles` (list[str]) All interactions will be structured in the following way, with the appropriate values filled in. [[ ## claim ## ]] {claim} [[ ## notes ## ]] {notes} [[ ## context ## ]] {context} [[ ## reasoning ## ]] {reasoning} [[ ## new_notes ## ]] {new_notes} # note: the value you produce must be pareseable according to the following JSON schema: {"type": "array", "items": {"type": "string"}} [[ ## titles ## ]] {titles} # note: the value you produce must be pareseable according to the following JSON schema: {"type": "array", "items": {"type": "string"}} [[ ## completed ## ]] In adhering to this structure, your objective is: Analyze the provided `claim`, `notes`, and `context` to generate `new_notes` that further support or refute the claim, and extract relevant `titles` that represent key topics or entities from the context. User message: This is an example of the task, though some input or output fields are not supplied. [[ ## claim ## ]] The figure who currently holds the role Michael Noonan held from 2001 to 2002, was the Minister for Social Protection from 2016 until 2017. [[ ## notes ## ]] Not supplied for this particular example. [[ ## context ## ]] Not supplied for this particular example. Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## new_notes ## ]]` (must be formatted as a valid Python list[str]), then `[[ ## titles ## ]]` (must be formatted as a valid Python list[str]), and then ending with the marker for `[[ ## completed ## ]]`. Assistant message: [[ ## reasoning ## ]] Not supplied for this particular example. [[ ## new_notes ## ]] Not supplied for this particular example. [[ ## titles ## ]] ["Michael Noonan", "Leader of Fine Gael", "Leo Varadkar"] [[ ## completed ## ]] User message: This is an example of the task, though some input or output fields are not supplied. [[ ## claim ## ]] The type of area Wiliwili are typically found in have a dominant tree species of Acacia koa. They are of the pea family. [[ ## notes ## ]] Not supplied for this particular example. [[ ## context ## ]] Not supplied for this particular example. Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## new_notes ## ]]` (must be formatted as a valid Python list[str]), then `[[ ## titles ## ]]` (must be formatted as a valid Python list[str]), and then ending with the marker for `[[ ## completed ## ]]`. Assistant message: [[ ## reasoning ## ]] Not supplied for this particular example. [[ ## new_notes ## ]] Not supplied for this particular example. [[ ## titles ## ]] ["Acacia koa", "Wiliwili", "Hawaiian tropical dry forests"] [[ ## completed ## ]] User message: [[ ## claim ## ]] The father of Zak Ové and A. Edward Sutherland are not both photographers. [[ ## notes ## ]] [1] «Horace Ové is a photographer, filmmaker, and writer.» [2] «A. Edward Sutherland is a film director, not a photographer.» [3] «Horace Ové is a photographer, filmmaker, and writer.» [4] «A. Edward Sutherland is a film director, not a photographer.» [5] «Horace Ové is a photographer, filmmaker, and writer.» [6] «A. Edward Sutherland is a film director, not a photographer.» [7] «Horace Ové is a photographer, filmmaker, and writer.» [8] «A. Edward Sutherland is a film director, not a photographer.» [[ ## context ## ]] {"Horace Ové | Horace Ové, CBE (born 1939), is a British filmmaker, photographer, painter and writer, one of the leading black independent film-makers to emerge in Britain since the post-war period. Ové holds the \"Guinness World Record\" for being the first black British film-maker to direct a feature-length film, \"Pressure\" (1975). In its retrospective history, \"100 Years of Cinema\", the British Film Institute (BFI) declared: \"Horace Ové is undoubtedly a pioneer in Black British history and his work provides a perspective on the Black experience in Britain.\"": 15.836545944213867, "Zak Ové | Zak Ové (born 1966) is a British visual artist who works between sculpture, film and photography, living in London, UK, and Trinidad. His themes reflect \"his documentation of and anthropological interest in diasporic and African history, specifically that which is explored through Trinidadian carnival.\" In work that is \"filtered through his own personal and cultural upbringing, with a black Trinidadian father and white Irish mother\", he has exhibited widely in Europe, the United States and Africa, participating in international museum shows in London, Dakar, Paris, Dubai, Prague, Berlin, Johannesburg, Bamako and New York City. His father is the filmmaker Horace Ové and his sister is the actress Indra Ové.": 13.145259857177734, "Playing Away | Playing Away is a 1987 TV comedy film directed by Horace Ové, from a screenplay by Caryl Phillips. In the story, an English cricket team, fictitiously named \"Sneddington\" (based in Lavenham, Suffolk), invites a team of West Indian heritage based in Brixton (South London) to play a charity game in support of their \"Third World Week.\" According to Screenonline, \"The gentle comedy of manners and unexpected reversal of white and black stereotypes in \"Playing Away\" contrasts sharply with the stylistic experimentation and the militant denunciations of racial prejudice in director Horace Ové's earlier feature, \"Pressure\" (1975).\" \" New York Times\" reviewer Vincent Canby called it \"witty and wise without being seriously disturbing for a minute\".": 12.445182800292969, "Pressure (film) | Pressure is a 1976 British drama film and the first feature-length fiction film directed by a Black film-maker in Britain. Directed by Horace Ové, and co-written by him with Samuel Selvon, \"Pressure\" is a powerful portrait of inter-generational tensions between first- and second-generation West Indian migrants in London's Notting Hill area. According to Julia Toppin,": 10.526924133300781, "What a Night! (1928 film) | What a Night! (1928) is an American silent film directed by A. Edward Sutherland. The romantic comedy was written by Louise Long, from a story by Lloyd Corrigan and Grover Jones. The film stars Bebe Daniels, Neil Hamilton, and William Austin.": 10.453304290771484, "The Orchid House (TV serial) | The Orchid House is a four-part television serial that first aired on British television's Channel 4 from 21 February to 14 March 1991, directed by Horace Ové. Its cast featured Diana Quick, Madge Sinclair, Nigel Terry, Elizabeth Hurley, Kate Buffery and Frances Barber, and was based on Phyllis Shand Allfrey's only novel of the same name.": 10.413471221923828, "Indra Ové | Indra Ové (born 1968) is a British movie and stage actress.": 10.15088176727295, "Henry Sutherland Edwards | Henry Sutherland Edwards (1828–1906) was a British journalist.": 9.868606567382812, "Burning an Illusion | Burning an Illusion is a 1981 British film written and directed by Menelik Shabazz, about a young British-born black woman's love life, mostly shot in London's Notting Hill and Ladbroke Grove communities. It was only the second British feature to have been made by a black director, following Horace Ové’s 1975 \"Pressure\", and is described by Stephen Bourne as \"the first British film to give a black woman a voice of any kind.\" Imruh Bakari worked with Shabazz and co-founded Kumba productions with him.": 9.634703636169434, "Horace Hall Edwards | Horace Hall Edwards (August 21, 1902 – January 27, 1987) was an American politician, who was mayor of Richmond, Virginia, and a candidate for governor in 1949.": 9.451831817626953} Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## new_notes ## ]]` (must be formatted as a valid Python list[str]), then `[[ ## titles ## ]]` (must be formatted as a valid Python list[str]), and then ending with the marker for `[[ ## completed ## ]]`. Assistant message: [[ ## reasoning ## ]] The claim states that the father of Zak Ové and A. Edward Sutherland are not both photographers. From the context, we know that Zak Ové's father is Horace Ové, who is indeed a photographer among other professions. However, A. Edward Sutherland is not mentioned as a photographer in the context; he is known as a film director. Therefore, the claim is correct because Horace Ové is a photographer, but A. Edward Sutherland is not. [[ ## new_notes ## ]] ["Horace Ové is a photographer, filmmaker, and writer.", "A. Edward Sutherland is a film director, not a photographer."] [[ ## titles ## ]] ["Horace Ové", "A. Edward Sutherland", "Zak Ové"] [[ ## completed ## ]] User message: [[ ## claim ## ]] The summer 2016 romantic drama "Me Before You" is directed by Thea Sharrock. The star of the film The Lost Future (who also appears in The Hunger Games) stars as the character Will Traynor. [[ ## notes ## ]] [1] «Sam Claflin stars as Will Traynor in 'Me Before You'.» [2] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.» [3] «Thea Sharrock directed 'Me Before You'.» [4] «Sam Claflin stars as Will Traynor in 'Me Before You'.» [5] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.» [6] «Thea Sharrock directed 'Me Before You'.» [7] «Sam Claflin starred in 'The Lost Future'.» [8] «Sam Claflin stars as Will Traynor in 'Me Before You'.» [9] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.» [10] «Thea Sharrock directed 'Me Before You'.» [11] «Sam Claflin starred in 'The Lost Future'.» [12] «Sam Claflin stars as Will Traynor in 'Me Before You'.» [13] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.» [14] «Thea Sharrock directed 'Me Before You'.» [15] «Sam Claflin starred in 'The Lost Future'.» [[ ## context ## ]] {"Sam Claflin | Samuel George Claflin (born 27 June 1986) is an English actor. He is known for portraying Finnick Odair in \"The Hunger Games\" film series, Philip Swift in \"\", and Will Traynor in \"Me Before You\".": 19.94539451599121, "Me Before You (film) | Me Before You is a 2016 romantic drama film directed by Thea Sharrock in her directorial debut and adapted by English author Jojo Moyes from her 2012 novel of the same name. The film stars Emilia Clarke, Sam Claflin, Steve Peacocke, Jenna Coleman, Charles Dance, Matthew Lewis, Janet McTeer, Vanessa Kirby and Joanna Lumley.": 18.48834228515625, "Look What You Did to Me | Look What You Did to Me is the debut studio album by American rapper and singer Z-Ro. It was released on June 16, 1998, by this independently distributed label Fisherboy Records. The album is entirely produced by Z-Ro alongside Rakish Jacob (aka Roc), while the guest appearances was from T.A.Z., Bam, Trae, The Fakkulty, Chris Ward and Al-D.": 14.100790977478027, "I Still Know What You Did Last Summer | I Still Know What You Did Last Summer is a 1998 American slasher film and a sequel to the 1997 film \"I Know What You Did Last Summer\". Directed by Danny Cannon, the film was written by Trey Callaway, and features characters originally created in Lois Duncan's 1973 novel \"I Know What You Did Last Summer\". Jennifer Love Hewitt, Freddie Prinze, Jr. and Muse Watson reprise their roles, with Brandy, Mekhi Phifer, Jennifer Esposito, and Matthew Settle joining the cast. \"I Still Know What You Did Last Summer\" continues after the events of the first film.": 13.990736961364746, "Tell Me What You Dream | \"Tell Me What You Dream\" is a song written by Timothy B. Schmit, Josh Leo and Vince Melamed and performed by country group Restless Heart along with saxophonist Warren Hill. The single was the group's only number one on the adult contemporary chart and despite previous country chart success, the song did not make the country top 40. \"Tell Me What You Dream\" spent two weeks at number one and peaked at number forty-three on the \"Billboard\" Hot 100.": 13.774335861206055, "Geraldine Jones (character) | Geraldine Jones was a fictional African American character, the most famous recurring persona of comedian Flip Wilson. Geraldine was played as a sassy liberated Southern woman who was coarsely flirty yet faithful to her (unseen) boyfriend \"Killer\". Poorly educated, she was nevertheless confident; she did not change her behavior to suit anyone. Several of Geraldine's sayings entered U.S. popular culture as catchphrases, especially \"When you're hot, you're hot; when you're not, you're not,\" \"The Devil made me do it,\" and \"What you see is what you get! \"": 13.149161338806152, "When Did You Stop Loving Me, When Did I Stop Loving You | When Did You Stop Loving Me, When Did I Stop Loving You": 12.946025848388672, "Pajama Sam 3: You Are What You Eat from Your Head to Your Feet | Pajama Sam 3: You Are What You Eat from Your Head to Your Feet": 12.687528610229492, "Sam Quartin | Sam Quartin is a musician, producer and actress best known for role as June Glass, starring alongside Marilyn Manson in the American movie Let Me Make You a Martyr\"(2017), also for \"By the Rivers of Babylon)\" and \"Aimy in a Cage\".": 12.639335632324219, "Why You Wanna Treat Me So Bad? | \"Why You Wanna Treat Me So Bad?\" is the US follow-up single to Prince's first big hit, \"I Wanna Be Your Lover\". It is also Prince's first rock and roll-inspired single release. It did not make the top 40 of the Billboard Hot 100 charts, although it did reach #13 on the R&B Singles charts. The lyrics are self-explanatory from the title, and deal with a cruel lover. The song prominently features guitar and bass, with the keyboard featured in a less prominent role. A highlight of the song is a soaring guitar solo at the end, played by Prince himself.": 12.574298858642578} Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## new_notes ## ]]` (must be formatted as a valid Python list[str]), then `[[ ## titles ## ]]` (must be formatted as a valid Python list[str]), and then ending with the marker for `[[ ## completed ## ]]`. Assistant message: [[ ## reasoning ## ]] The claim states that the film "Me Before You" is directed by Thea Sharrock and that the actor who stars as Will Traynor also appeared in "The Lost Future" and "The Hunger Games". According to the notes, Sam Claflin stars as Will Traynor in "Me Before You" and is known for his role as Finnick Odair in "The Hunger Games". Additionally, Sam Claflin starred in "The Lost Future". The context confirms that "Me Before You" is directed by Thea Sharrock and that Sam Claflin plays Will Traynor. Therefore, the claim is supported by the provided notes and context. [[ ## new_notes ## ]] ["Sam Claflin stars as Will Traynor in 'Me Before You'.", "Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.", "Thea Sharrock directed 'Me Before You'.", "Sam Claflin starred in 'The Lost Future'."] [[ ## titles ## ]] ["Me Before You (film)", "Sam Claflin", "Thea Sharrock", "The Lost Future", "The Hunger Games"] [[ ## completed ## ]] User message: [[ ## claim ## ]] The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights. [[ ## notes ## ]] N/A [[ ## context ## ]] {"Up Against It | Up Against It is an unproduced script by Joe Orton, written in 1967 for The Beatles at the height of their fame.": 23.741416931152344, "The Alien (unproduced film) | The Alien was an unproduced Indian-American science fiction film in development in the late 1960s which was eventually cancelled. It was to be directed by celebrated Indian filmmaker Satyajit Ray and co-produced by Columbia Pictures. The script was written by Ray in 1967, loosely based on Bankubabur Bandhu (\"Banku Babu's Friend\" or \"Mr. Banku's Friend\"), a Bengali science fiction story he had written in 1962 for \"Sandesh\", the Ray family magazine, which gained popularity among Bengalis in the early 1960s. \"Bankubabur Bandhu\" was eventually adapted into a television film by Satyajit Ray's son Sandip Ray, and a play by the theatre group Swapnasandhani Kaushik Sen, in 2006.": 14.683004379272461, "Marsha Albert | Marsha Albert (born 1948) is credited with being the person who jump-started the early 1960s phenomena known as Beatlemania in the United States when as a 15-year-old girl, on 17 December 1963, she introduced for the first time on American radio a song written and recorded by The Beatles titled I Want to Hold Your Hand (the Beatles' best-selling single worldwide), and that Beatles historian and author Bruce Spizer noted, in 2004, by his stating \"Marsha Albert's actions forced a major record company to push up the release date of a debut single from an unknown band during the holiday season, a time when record companies traditionally released no new product.\"": 13.236483573913574, "Up Against It! (Todd Rundgren album) | Up Against It! is a 1997 album by Todd Rundgren, essentially consisting of Rundgren's song demos for the Off Broadway show that were Written and Recorded by Todd from 1986-88. The project was inspired by the never-produced Up Against It which was a play originally written by Joe Orton for The Beatles.": 12.989564895629883, "Blood and Fire (Star Trek: The Next Generation) | \"Blood and Fire\" is an episode written by David Gerrold for possible use on \"\". The script was commissioned and written, but never actually filmed. According to Gerrold, some of the production staff, including Rick Berman, had a negative reaction to its positive depiction of an openly gay couple. Herbert Wright rewrote the script under the name \"Blood and Ice\", which also was left unproduced.": 11.980508804321289, "Cedar Rapids (film) | Cedar Rapids is a 2011 American comedy film directed by Miguel Arteta. The script, written by Phil Johnston, was included on the 2009 Black List, a Hollywood list of the most popular unproduced screenplays of the year.": 11.593443870544434, "The Beatles: The Biography | The Beatles: The Biography is the name of a 2005 biography of the 1960s rock band The Beatles written by Bob Spitz. It was first published by Little, Brown and Company on November 1, 2005.": 11.583497047424316, "The Illusionist (2010 film) | The Illusionist (French: L'Illusionniste ) is a 2010 French-British animated film directed by Sylvain Chomet. The film is based on an unproduced script written by French mime, director and actor Jacques Tati in 1956. Controversy surrounds Tati's motivation for the script, which was written as a personal letter to his estranged eldest daughter, Helga Marie-Jeanne Schiel in collaboration with his long-term writing partner Henri Marquet, between writing for the films \"Mon Oncle\" and \"Play Time\".": 11.528533935546875, "Bobby Florsheim | Robert Florsheim (born December 19, 1969) is an American screenwriter, best known for co-writing \"The Passion Of The Ark\" with Josh Stolberg, (the basis for the film \"Evan Almighty\"). Their original script was reported by Daily Variety as the highest priced spec script ever sold by unproduced screenwriters (April, 2004). In refashioning the script into \"Evan Almighty\", Universal Studios discarded the script and then hired screenwriter Steve Oedekerk, who received sole writing credit on the finished film.": 11.23757553100586, "The Hook (screenplay) | The Hook is an unproduced screenplay by American playwright, Arthur Miller. It was written in 1947 and was intended to be produced by Columbia Pictures Studio, Hollywood, and to be directed by Elia Kazan. The screenplay was inspired by the true story of Pete Panto, a young dockworker who stood up against the corrupt Mafia-connected union leadership. Panto was discovered dead in a pit outside New York eighteen months after his disappearance. Set in the Red Hook district of Brooklyn, \"The Hook\" is the story of Marty Ferrara, a longshoreman who is ‘ready to lay down his life, if need be, to secure one thing – his sense of personal dignity.’": 11.222980499267578} Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## new_notes ## ]]` (must be formatted as a valid Python list[str]), then `[[ ## titles ## ]]` (must be formatted as a valid Python list[str]), and then ending with the marker for `[[ ## completed ## ]]`. Response: [[ ## reasoning ## ]] The claim states that the author of the 1960s unproduced script written for The Beatles, "Up Against It", and Bernard-Marie Koltès are both playwrights. The context confirms that Joe Orton wrote the script "Up Against It" for The Beatles, and Bernard-Marie Koltès was a French playwright. Therefore, the claim is supported by the provided context. [[ ## new_notes ## ]] [] [[ ## titles ## ]] ["Up Against It", "Bernard-Marie Koltès", "Joe Orton", "The Beatles"] [[ ## completed ## ]]
最后，保存我们优化的程序，以便以后再次使用。
输入 [16]
已复制！
optimized.save("optimized_hop.json")
loaded_program = Hop()
loaded_program.load("optimized_hop.json")
loaded_program(claim="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles
optimized.save("optimized_hop.json") loaded_program = Hop() loaded_program.load("optimized_hop.json") loaded_program(claim="为披头士乐队编写的 20 世纪 60 年代未制作的剧本《Up Against It》的作者和 Bernard-Marie Koltès 都是剧作家。").titles
输出[16]
['Up Against It', 'Bernard-Marie Koltès', 'The Beatles', 'Joe Orton']
在 MLflow 实验中保存程序
除了将程序保存到本地文件，您还可以在 MLflow 中跟踪它，以获得更好的可重现性和协作性。
- 依赖管理：MLflow 会自动保存冻结的环境元数据以及程序，以确保可重现性。
- 实验跟踪：使用 MLflow，您可以跟踪程序的性能和成本以及程序本身。
- 协作：您可以通过共享 MLflow 实验与团队成员共享程序和结果。
要在 MLflow 中保存程序，运行以下代码
import mlflow
# Start an MLflow Run and save the program
with mlflow.start_run(run_name="optimized"):
model_info = mlflow.dspy.log_model(
optimized,
artifact_path="model", # Any name to save the program in MLflow
)
# Load the program back from MLflow
loaded = mlflow.dspy.load_model(model_info.model_uri)
要了解更多关于集成的信息，请访问MLflow DSPy 文档。