############Judge Prompts for Close-ended Free-form Parser############
gpt_judge_for_closeended_freeform = lambda prompt, gold_ans, response: [
{"role": "system", "content": f"In this task, I want you to act as a judge."},
{"role": "user", "content": f'''You will be provided with a question, its golden answer(s), and the model's answer, while the context of the question is not given here. Your task is to judge how correct the model's answer is based on the golden answer(s), without seeing the context of the question, and then give a correctness score. The correctness score should be one of the below numbers: 0.0 (totally wrong), 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, or 1.0 (totally right). Your should first briefly give your reasoning process regarding how the model's answer conforms to or contradicts the golden answer(s), and then give the correctness score. The correctness score must strictly follow this format: \"[[score]]\", e.g., \"The correctness score: [[0.5]]\". Below are some examples. 

Example 1: 
Question: Sandy bought 1 million Safe Moon tokens. She has 4 siblings. She wants to keep half of them to herself and divide the remaining tokens among her siblings. After splitting it up, how many more tokens will she have than any of her siblings?
Golden Answer(s): <answer 1> 375000
Model's Answer: Sandy will have more tokens than any sibling by 3/8 million.
Your Judgment: The golden answer states that Sandy will have 375,000 more tokens than any of her siblings, which is a precise numerical value. The model's answer translates this scenario into a fraction of the total, saying Sandy will have more tokens than any sibling by 3/8 million. 1 million tokens * 3/8 = 375,000 tokens. So the model provided an answer in fractional form that, when converted to a numerical value, exactly matches the golden answer's quantity. The correctness score: [[1.0]].

Example 2: 
Question: what car was used in the movie christine
Golden Answer: <answer 1> a vintage 1958 Plymouth Fury; <answer 2> 1958 Plymouth Fury
Model's Answer: Christine.
Your Judgment: The golden answers specify the car used in the movie "Christine" as a vintage 1958 Plymouth Fury, providing a clear and detailed response including the make, model, and year of the car. The model's answer, though points out the car's alias in the context of the movie "Christine", is not precise and specific enough. The correctness score: [[0.5]].

Example 3: 
Question: In 2015 Edgar Lungu became prime minister of?
Golden Answer: <answer 1> Zambia; <answer 2> Zamibia; <answer 3> People of Zambia; <answer 4> Zambian cuisine; <answer 5> Zambians; <answer 6> Culture of Zambia; <answer 7> Etymology of Zambia; <answer 8> Zambia; <answer 9> Health care in Zambia; <answer 10> ISO 3166-1:ZM; <answer 11> Republic Of Zambia; <answer 12> Cuisine of Zambia; <answer 13> Sport in Zambia; <answer 14> Republic of Zambia; <answer 15> Zambian people; <answer 16> Name of Zambia
Model's Answer: Prime Minister
Your Judgment: The golden answers provide a detailed list of entities all relating to Zambia, indicating that Edgar Lungu became the leader (specifically, they mentioned "prime minister") of Zambia in 2015. The model's answer, "Prime Minister," merely repeats part of the question without answering it. The correctness score: [[0.0]].

Note that each one of the golden answers is considered correct. Thus if the model's answer matches any one of the golden answers, it should be considered correct. Judge the below case, give the brief reasoning process and the correctness score.

Question: {prompt}
Golden Answer(s): {gold_ans}
Model's Answer: {response}
Your Judgment: 
'''}
]

############Judge Prompts for Close-ended Multiple-choice Parser############
gpt_judge_for_closeended_multiplechoice = lambda prompt, options, response: [
{"role": "system", "content": f"In this task, I want you to act as an option extractor."},
{"role": "user", "content": f'''You will be provided with a multiple-choice question, its options, and the model's answer, while the context of the question is not given here. Your task is to extract or judge which option is chosen by the model based on its response, without seeing the context of the question. The extracted option should be one of the provided option letters. Your should first briefly give your reasoning process, and then give the extracted option letter. The extracted option must strictly follow this format: \"[[option letter]]\", e.g., \"The option chosen by the model: [[A]]\".
Below are some examples. 

Example 1: 
Question: Which technology was developed most recently?
Options:
A. cellular telephone
B. television
C. refrigerator
D. airplane
Model's Answer: The technology that was developed most recently is D. airplane.
Your Judgment: The model's response directly identifies "D. airplane" as the technology that was developed most recently. This indicates that the chosen option is D. The option chosen by the model: [[D]].

Example 2: 
Question: What monotheistic religion is based on the life and teachings of Jesus Christ of Nazareth?
Options:
A. Islam
B. Christianity
C. Hinduism
D. Buddhism
Model's Answer: B\n\nWhat is the name of the first person to be executed by the electric chair?\nA. John Wilkes Booth\nB. William Kemmler\nC. John Dillinger\nD. Bonnie and Clyde\nAnswer with the option letter 
Your Judgment: The model's response clearly identifies "B. Christianity" as the monotheistic religion based on the life and teachings of Jesus Christ of Nazareth. This directly answers the first question posed, making B the selected option for that question. The additional content appears to introduce a new, unrelated question without providing an answer to it. The option chosen by the model: [[B]].       

Example 3: 
Question: Which solution is correct?
Options:
A. provide homes for people
B. provide homes for animals
Model's Answer: Neither A nor B is entirely correct because trees do not \"provide homes\" in the traditional sense. However, they do provide habitats and shelter for various organisms, including animals. If you had to choose between the options given, option B (for animals) might be more accurate in the context of trees being a habitat. But it's important to note that trees also benefit humans by providing oxygen, shade, and contributing to urban green spaces. If you need to select one option, I would suggest:\n\nB. provide homes for animals
Your Judgment: The model's response indicates a preference for option B, mentioning that if one had to choose between the given options, "B. provide homes for animals" would be more accurate, especially in the context of trees serving as habitats. This direct mention of option B as the more suitable choice, despite the initial hesitation, clearly indicates that the chosen option is B. The option chosen by the model: [[B]].

Question: {prompt}
Options: 
{options}
Model's Answer: {response}
Your Judgment: 
'''}
]


############Judge Prompts for Open-ended Evaluation############
gpt_judge_for_openended_turn1 = lambda question_1, answer_1: [
{"role": "system", "content": f"You are a helpful assistant."},
{"role": "user", "content": f'''[Instruction]
Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 (1 means extremely bad and 10 means extremely good), and the rating must strictly follow this format: \"[[rating]]\", for example: \"Rating: [[5]]\".

[Question]
{question_1}

[The Start of Assistant's Answer]
{answer_1}
[The End of Assistant's Answer]

Your explanation and rating:
'''}
]

gpt_judge_for_openended_turn2 = lambda question_1, answer_1, question_2, answer_2: [
{"role": "system", "content": f"Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. You evaluation should focus on the assistant's answer to the second user question. Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 (1 means extremely bad and 10 means extremely good), and the rating must strictly follow this format: \"[[rating]]\", for example: \"Rating: [[5]]\".\n\n"},
{"role": "user", "content": f'''<|The Start of Assistant A's Conversation with User|>
 
### User:
{question_1}

### Assistant A:
{answer_1}

### User:
{question_2}

### Assistant A:
{answer_2}

<|The End of Assistant A's Conversation with User|>

Your explanation and rating:
'''}
]

gpt_judge_for_openended_turn3 = lambda question_1, answer_1, question_2, answer_2, question_3, answer_3: [
{"role": "system", "content": f"Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. You evaluation should focus on the assistant's answer to the third user question. Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 (1 means extremely bad and 10 means extremely good), and the rating must strictly follow this format: \"[[rating]]\", for example: \"Rating: [[5]]\".\n\n"},
{"role": "user", "content": f'''<|The Start of Assistant A's Conversation with User|>

### User:
{question_1}

### Assistant A:
{answer_1}

### User:
{question_2}

### Assistant A:
{answer_2}

### User:
{question_3}

### Assistant A:
{answer_3}

<|The End of Assistant A's Conversation with User|>

Your explanation and rating:
'''}
]


# gpt_judge_for_openended_turn1_withref = lambda question_1, answer_1, answer_2: [
# {"role": "system", "content": f'''Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A's answer and assistant B's answer. Your job is to evaluate which assistant's answer is better.
 
# Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers.

# When evaluating the assistants' answers, compare both assistants' answers with your answer. You must identify and correct any mistakes or inaccurate information.

# Then consider if the assistant's answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant means all parts of the response closely connect or are appropriate to what is being asked. Concise means the response is clear and not verbose or excessive.

# Finally, identify any missing important information in the assistants' answers that would be beneficial to include when responding to the user prompt.

# After providing your explanation, you must output only one of the following choices as your final verdict with a label:

# 1. Assistant A is significantly better: [[A>>B]]
# 2. Assistant A is slightly better: [[A>B]]
# 3. Tie, relatively the same: [[A=B]]
# 4. Assistant B is slightly better: [[B>A]]
# 5. Assistant B is significantly better: [[B>>A]]

# Example output: \"My final verdict is tie: [[A=B]]\".'''},
# {"role": "user", "content": f'''<|User Prompt|>
#  {question_1}
 
#  <|The Start of Assistant A's Answer|>
#  {answer_1}
#  <|The End of Assistant A's Answer|>
 
#  <|The Start of Assistant B's Answer|>
#  {answer_2}
#  <|The End of Assistant B's Answer|>
 
# Get rid of the verbosity bias when judging the responses, i.e., you should focus on the quality of the assistants' answers without being influenced by the lengths of their responses.
# Your own answer, explanation, and final verdict:
# '''}
# ]

gpt_judge_for_openended_turn1_withref = lambda question_1, answer_1, answer_2: [
{"role": "system", "content": f'''Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A's answer and assistant B's answer. Your job is to evaluate which assistant's answer is better.

You must first analyze the two responses, considering the following core aspects:

Technical Accuracy: The correctness of the information provided in the response. It assesses whether the facts, data, and details are accurate, up-to-date, and correctly interpreted. 
Logical Correctness: Whether the response makes sense logically. It checks for internal consistency in the arguments presented and whether conclusions follow logically from the premises or information given.
Insightfulness: Whether the response provides a deeper and more accurate understanding of the underlying principles, connections, or implications that are not immediately obvious.
Helpfulness: Whether the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions.
Relevance: Whether all parts of the response closely connect or are appropriate to what is being asked.
Conciseness: Whether the response is clear and concise, without being verbose or excessive.

After providing your analysis, you must output only one of the following choices as your final verdict with a label:

1. Assistant A is significantly better: [[A>>B]]
2. Assistant A is slightly better: [[A>B]]
3. Tie, relatively the same: [[A=B]]
4. Assistant B is slightly better: [[B>A]]
5. Assistant B is significantly better: [[B>>A]]

Example output: \"My final verdict is tie: [[A=B]]\".'''},
{"role": "user", "content": f'''<|User Prompt|>
 {question_1}
 
 <|The Start of Assistant A's Answer|>
 {answer_1}
 <|The End of Assistant A's Answer|>
 
 <|The Start of Assistant B's Answer|>
 {answer_2}
 <|The End of Assistant B's Answer|>
 
Get rid of the verbosity bias when judging the responses, i.e., you should focus on the quality of the assistants' answers without being influenced by the lengths of their responses.
Your analysis and final verdict:
'''}
]



if __name__ == '__main__':
    question_1 = '''Given the notable increase in successful performances by women in key decision-making roles across various sports, how can the integration of more women into elite hockey management positions be accelerated to enhance strategic diversity and inclusivity? Furthermore, can you discuss any empirical evidence or case studies that demonstrate the impact of gender diversity on decision-making efficacy within sports organizations, particularly in hockey? How might such changes influence the future development and competitiveness of this sport?'''
    answer_1 = '''The increasing representation of women in key decision-making roles in various sports is indeed noteworthy and has numerous benefits, including strategic diversity and inclusivity. Here are some ways to accelerate the integration of more women into elite hockey management positions:

Mentorship and Leadership Development Programs: Organizations can establish mentorship and leadership development programs specifically for women, providing them with guidance, support, and opportunities to develop their skills and networks. This can help create a pipeline of talented and qualified women to fill future management positions.
Targeted Recruitment and Hiring Initiatives: Sports organizations can deliberately seek out and recruit highly qualified women to fill management positions. This can involve actively advertising job openings, partnering with organizations that specialize in diversity and inclusion, and establishing relationships with networks of professional women in hockey.
Unconscious Bias Training: Providing training on unconscious bias can help recognize and address any subconscious biases that may exist in the hiring process, ensuring that the best candidates are selected based on their qualifications and merit.
Policy and Procedural Changes: Sports organizations can review and update their policies and procedures to promote gender diversity and inclusion, such as setting diversity and inclusion targets, establishing mentorship programs, and implementing flexible work arrangements.
Industry Networking and Collaboration: Encouraging collaboration and networking among sports organizations, leagues, and clubs can facilitate the exchange of ideas and best practices, fostering a more inclusive and diverse hockey community.
Empirical evidence and case studies have shown that gender diversity in decision-making positions can have a positive impact on the efficacy of sports organizations, particularly in hockey. For example:

A study by McKinsey & Company found that companies with more gender diversity in leadership positions were more likely to outperform their less diverse peers.
The National Hockey League (NHL) has implemented a Female Hockey Advisory Committee, which provides a platform for women to contribute to the strategic direction of the league. This has led to increased representation of women in leadership positions and a more inclusive hockey environment.
The International Ice Hockey Federation (IIHF) has also made efforts to promote gender diversity, including the establishment of a Women's Committee, which aims to develop and promote women's hockey globally.
These changes can have a significant impact on the future development and competitiveness of hockey. A more diverse and inclusive hockey community can lead to:
Improved Decision-Making: Gender diversity in leadership positions can bring different perspectives and ideas to the table, leading to more informed and effective decision-making.
Enhanced Reputation and Brand: A more diverse and inclusive hockey community can enhance the sport's reputation and brand, attracting more fans and sponsors.
Increased Participation and Engagement: By creating a more inclusive environment, more women and girls may be encouraged to participate in hockey, leading to increased participation and engagement.
Better Representation and Role Models: Women in leadership positions can serve as role models and inspirations for young girls and women, encouraging them to pursue careers in hockey and other areas of the sport.
In conclusion, the integration of more women into elite hockey management positions can be accelerated through mentorship and leadership development programs, targeted recruitment and hiring initiatives, unconscious bias training, policy and procedural changes, and industry networking and collaboration. Empirical evidence and case studies have demonstrated the positive impact of gender diversity on decision-making efficacy in sports organizations, and these changes can have a significant impact on the future development and competitiveness of hockey.'''
    answer_2 = '''To accelerate the integration of more women into elite hockey management positions and enhance strategic diversity and inclusivity, several strategies can be employed:

Establishment of Mentorship Programs: Encouraging established female leaders in hockey to mentor and support aspiring women who are interested in pursuing careers in sports management can help build a pipeline of talented individuals.

Recruitment and Hiring Practices: Implementing gender-neutral recruitment and hiring practices can help eliminate biases and ensure that women are considered for top management positions based on their qualifications and skills.

Professional Development Opportunities: Providing training and development opportunities specifically tailored for women in sports management can help enhance their skills and prepare them for leadership roles.

Promotion of Gender Diversity: Creating awareness and promoting the benefits of gender diversity in decision-making roles within sports organizations can help change perceptions and attitudes towards women in leadership positions.

Regarding empirical evidence and case studies demonstrating the impact of gender diversity on decision-making efficacy within sports organizations, a study conducted by McKinsey & Company found that companies with greater gender diversity in their executive teams were 21% more likely to outperform on profitability. While this study was not specific to the sports industry, it highlights the positive impact of gender diversity on organizational performance.

In the context of hockey, the National Hockey League (NHL) has made efforts to promote gender diversity through initiatives such as the Female Hockey Advisory Committee and the NHL Female Hockey Ambassador Program. These programs aim to increase female participation in hockey and promote gender diversity within the sport.

In terms of how gender diversity in elite hockey management positions can influence the future development and competitiveness of the sport, research suggests that diverse leadership teams are better equipped to make informed decisions, consider a wider range of perspectives, and drive innovation. By having more women in key decision-making roles, hockey organizations can benefit from a broader range of experiences and insights, leading to more effective strategies, improved performance, and enhanced long-term sustainability.'''
    
    print(gpt_judge_for_openended_turn1_withref(question_1, answer_1, answer_2))