面向开发者的LLM入门课程-不存在简单的正确答案英文版

哈喽！伙伴们，我是小智，你们的AI向导。欢迎来到每日的AI学习时间。今天，我们将一起深入AI的奇妙世界，探索“面向开发者的LLM入门课程-不存在简单的正确答案英文版”，并学会本篇文章中所讲的全部知识点。还是那句话“不必远征未知，只需唤醒你的潜能！”跟着小智的步伐，我们终将学有所成，学以致用，并发现自身的更多可能性。话不多说，现在就让我们开始这场激发潜能的AI学习之旅吧。

面向开发者的LLM入门课程-不存在简单的正确答案英文版：

英文版

1.对问答系统提问

import utils_en
# 用户消息
customer_msg = f”””
tell me about the smartx pro phone and the fotosnap camera, the dslr one.
Also, what TVs or TV related products do you have?”””
# 从问题中抽取商品名
products_by_category = utils_en.get_products_from_query(customer_msg)
# 将商品名转化为列表
category_and_product_list = utils_en.read_string_to_list(products_by_category)
# 查找商品对应的信息
product_info = utils_en.get_mentioned_product_info(category_and_product_list)
# 由信息生成回答
assistant_answer = utils_en.answer_user_msg(user_msg=customer_msg,
product_info=product_info)

print(assistant_answer)

Sure! Let me provide you with some information about the SmartX ProPhone and the
FotoSnap DSLR Camera.
The SmartX ProPhone is a powerful smartphone with advanced camera features. It
has a 6.1-inch display, 128GB storage, a 12MP dual camera, and supports 5G
connectivity. The SmartX ProPhone is priced at $899.99 and comes with a 1-year
warranty.
The FotoSnap DSLR Camera is a versatile camera that allows you to capture
stunning photos and videos. It features a 24.2MP sensor, 1080p video recording, a
3-inch LCD screen, and supports interchangeable lenses. The FotoSnap DSLR Camera
is priced at $599.99 and also comes with a 1-year warranty.
As for TVs and TV-related products, we have a range of options available. Some of
our popular TV models include the CineView 4K TV, CineView 8K TV, and CineView
OLED TV. We also have home theater systems like the SoundMax Home Theater and
SoundMax Soundbar. Could you please let me know your specific requirements or
preferences so that I can assist you better?

2.使用GPT评估

# 问题、上下文
cust_prod_info = {
‘customer_msg’: customer_msg,
‘context’: product_info
}

def eval_with_rubric(test_set, assistant_answer):
“””
使用 GPT API 评估生成的回答
参数：
test_set: 测试集
assistant_answer: 助手的回复
“””
cust_msg = test_set[‘customer_msg’]
context = test_set[‘context’]
completion = assistant_answer
# 要求 GPT 作为一个助手评估回答正确性
system_message = “””
You are an assistant that evaluates how well the customer service agent
answers a user question by looking at the context that the customer service
agent is using to generate its response.
“””
# 具体指令
user_message = f”””
You are evaluating a submitted answer to a question based on the context
that the agent uses to answer the question.
Here is the data:
[BEGIN DATA]
************
[Question]: {cust_msg}
************
[Context]: {context}
************
[Submission]: {completion}
************
[END DATA]
Compare the factual content of the submitted answer with the context.
Ignore any differences in style, grammar, or punctuation.
Answer the following questions:
– Is the Assistant response based only on the context provided? (Y or N)
– Does the answer include information that is not provided in the context? (Y
or N)
– Is there any disagreement between the response and the context? (Y or N)
– Count how many questions the user asked. (output a number)
– For each question that the user asked, is there a corresponding answer to
it?
Question 1: (Y or N)
Question 2: (Y or N)
…
Question N: (Y or N)
– Of the number of questions asked, how many of these questions were
addressed by the answer? (output a number)
“””
messages = [
{‘role’: ‘system’, ‘content’: system_message},
{‘role’: ‘user’, ‘content’: user_message}
]
response = get_completion_from_messages(messages)
return response

evaluation_output = eval_with_rubric(cust_prod_info, assistant_answer)
print(evaluation_output)

– Is the Assistant response based only on the context provided? (Y or N)
Y
– Does the answer include information that is not provided in the context? (Y or
N)
N
– Is there any disagreement between the response and the context? (Y or N)
N
– Count how many questions the user asked. (output a number)
2
– For each question that the user asked, is there a corresponding answer to it?
Question 1: Y
Question 2: Y
– Of the number of questions asked, how many of these questions were addressed by
the answer? (output a number)
2

3.评估生成回答与标准回答的差距

test_set_ideal = {
‘customer_msg’: “””
tell me about the smartx pro phone and the fotosnap camera, the dslr one.
Also, what TVs or TV related products do you have?”””,
‘ideal_answer’:”””
Of course! The SmartX ProPhone is a powerful
smartphone with advanced camera features.
For instance, it has a 12MP dual camera.
Other features include 5G wireless and 128GB storage.
It also has a 6.1-inch display. The price is $899.99.
The FotoSnap DSLR Camera is great for
capturing stunning photos and videos.
Some features include 1080p video,
3-inch LCD, a 24.2MP sensor,
and interchangeable lenses.
The price is 599.99.
For TVs and TV related products, we offer 3 TVs
All TVs offer HDR and Smart TV.
The CineView 4K TV has vibrant colors and smart features.
Some of these features include a 55-inch display,
‘4K resolution. It’s priced at 599.
The CineView 8K TV is a stunning 8K TV.
Some features include a 65-inch display and
8K resolution. It’s priced at 2999.99
The CineView OLED TV lets you experience vibrant colors.
Some features include a 55-inch display and 4K resolution.
It’s priced at 1499.99.
We also offer 2 home theater products, both which include bluetooth.
The SoundMax Home Theater is a powerful home theater system for
an immmersive audio experience.
Its features include 5.1 channel, 1000W output, and wireless subwoofer.
It’s priced at 399.99.
The SoundMax Soundbar is a sleek and powerful soundbar.
It’s features include 2.1 channel, 300W output, and wireless subwoofer.
It’s priced at 199.99
Are there any questions additional you may have about these products
that you mentioned here?
Or may do you have other questions I can help you with?
“””
}

def eval_vs_ideal(test_set, assistant_answer):
“””
评估回复是否与理想答案匹配
参数：
test_set: 测试集
assistant_answer: 助手的回复
“””
cust_msg = test_set[‘customer_msg’]
ideal = test_set[‘ideal_answer’]
completion = assistant_answer
system_message = “””
You are an assistant that evaluates how well the customer service agent
answers a user question by comparing the response to the ideal (expert)
response
Output a single letter and nothing else.
“””
user_message = f”””
You are comparing a submitted answer to an expert answer on a given question.
Here is the data:
[BEGIN DATA]
************
[Question]: {cust_msg}
************
[Expert]: {ideal}
************
[Submission]: {completion}
************
[END DATA]
Compare the factual content of the submitted answer with the expert answer.
Ignore any differences in style, grammar, or punctuation.
The submitted answer may either be a subset or superset of the expert answer,
or it may conflict with it. Determine which case applies.
Answer the question by selecting one of the following options:
(A) The submitted answer is a subset of the expert answer and is fully
consistent with it.
(B) The submitted answer is a superset of the expert answer and is fully
consistent with it.
(C) The submitted answer contains all the same details as the expert answer.
(D) There is a disagreement between the submitted answer and the expert
answer.
(E) The answers differ, but these differences don’t matter from the
perspective of factuality.
choice_strings: ABCDE
“””
messages = [
{‘role’: ‘system’, ‘content’: system_message},
{‘role’: ‘user’, ‘content’: user_message}
]
response = get_completion_from_messages(messages)
return response

print(assistant_answer)

Sure! Let me provide you with some information about the SmartX ProPhone and the
FotoSnap DSLR Camera.
The SmartX ProPhone is a powerful smartphone with advanced camera features. It
has a 6.1-inch display, 128GB storage, a 12MP dual camera, and supports 5G
connectivity. The SmartX ProPhone is priced at $899.99 and comes with a 1-year
warranty.
The FotoSnap DSLR Camera is a versatile camera that allows you to capture
stunning photos and videos. It features a 24.2MP sensor, 1080p video recording, a
3-inch LCD screen, and supports interchangeable lenses. The FotoSnap DSLR Camera
is priced at $599.99 and also comes with a 1-year warranty.
As for TVs and TV-related products, we have a range of options available. Some of
our popular TV models include the CineView 4K TV, CineView 8K TV, and CineView
OLED TV. We also have home theater systems like the SoundMax Home Theater and
SoundMax Soundbar. Could you please let me know your specific requirements or
preferences so that I can assist you better?

# 由于模型的更新，目前在原有 Prompt 上不再能够正确判断
eval_vs_ideal(test_set_ideal, assistant_answer)

‘D’

assistant_answer_2 = “life is like a box of chocolates”

eval_vs_ideal(test_set_ideal, assistant_answer_2)
# 对于明显异常答案，GPT 判断为不一致

‘D’

嘿，伙伴们，今天我们的AI探索之旅已经圆满结束。关于“面向开发者的LLM入门课程-不存在简单的正确答案英文版”的内容已经分享给大家了。感谢你们的陪伴，希望这次旅程让你对AI能够更了解、更喜欢。谨记，精准提问是解锁AI潜能的钥匙哦！如果有小伙伴想要了解学习更多的AI知识，请关注我们的官网“AI智研社”，保证让你收获满满呦！