@@ -38,7 +38,7 @@ class Llama3_70B_ZeroShotModel(ShopBenchBaseModel):
self.temperature=0.0
self.exampler_generation="""
You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. If your answer is correct, I will give you a $200 tip.
You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. Please think step by step but only output final answers according to the format requirements of the question. If your answer is correct, I will give you a $200 tip.
Question:Instructions: Tell me what this product category is about\nInput: Toggle Switch\nOutput: A toggle switch is an electric switch operated by means of a projecting lever that is moved up and down.\n
...
...
@@ -58,7 +58,7 @@ class Llama3_70B_ZeroShotModel(ShopBenchBaseModel):
"""
self.exampler_multi=(
"You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. If your answer is correct, I will give you a $200 tip.\n\n"
"You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. Please think step by step but only output final answers according to the format requirements of the question. If your answer is correct, I will give you a $200 tip.\n\n"
"Question: Which of the following statements best describes the relation from query \"waterpik\" to query \"long sleeve crop tops for women\"?\nA. irrelevant\nB. substitute\nC. complement\nD. narrowing\nAnswer: A\n"
"Question: Which of the following statements best describes the relation from query \"bioderma\" to query \"bioderma eye cream\"?\nA. irrelevant\nB. substitute\nC. complement\nD. narrowing\nAnswer: D\n"
"Question: Which of the following statements best describes the relation from query \"kitkat\" to query \"cookies\"?\nA. narrowing\nB. substitute\nC. irrelevant\nD. complement\nAnswer: D\n"
...
...
@@ -70,7 +70,7 @@ class Llama3_70B_ZeroShotModel(ShopBenchBaseModel):
"Question: ")
self.exampler_multi15=(
"You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. If your answer is correct, I will give you a $200 tip.\n\n"
"You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. Please think step by step but only output final answers according to the format requirements of the question. If your answer is correct, I will give you a $200 tip.\n\n"
"Question: Instructions: Evaluate the following product review on a scale of 1 to 5, with 1 being very negative and 5 being very positive. \nInput: a review text for a(n) Leotard product\nRuns small and fabric doesn\u2019t stretch\nThis body shaper is way too small even though I bought 2X. It doesn\u2019t stretch at all to be form fitting. I don\u2019t like this product and will never purchase it again.\nOutput:\nAnswer: 1\n"
"Question: Instructions: Evaluate the following product review on a scale of 1 to 5, with 1 being very negative and 5 being very positive. \nInput: a review text for a(n) Boxing Glove product\nGood choice for begginers\/intermediates\nGloves look exactly like on a picture, love them. Good gloves for good price, especially for beginners. I used to using leather gloves and that leather smell was really bad, I am glad that this time I decided to buy synthetic. Unfortunally this may be con too, because durabality is lower. Anyway I prefer this synthetic ones. Also part on palms is made with textile for ventilation, so hands don't sweat that much.\nOutput:\nAnswer: 4\n"
"Question: Instructions: Evaluate the following product review on a scale of 1 to 5, with 1 being very negative and 5 being very positive. \nInput: a review text for a(n) Thermometer product\nHas that cheaply made feel\nWhile I believe the thermometer to be accurate, it has the "made in China" cheaply made feel. Additionally, it is larger than I expected. I was hoping it would fit in one of the thermometer and pen pockets on the arm of my chef jacket, but it is too large and does not fit. I have a number of the Taylor thermometers that are much better quality.\nOutput:\nAnswer: 3\n"
...
...
@@ -82,7 +82,7 @@ class Llama3_70B_ZeroShotModel(ShopBenchBaseModel):
"Question: ")
self.exampler_ranking=(
"You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. If your answer is correct, I will give you a $200 tip.\n\n"
"You are a very intelligent and helpful online shopping assistant for Amazon who can solve the following rating questions. Please think step by step but only output final answers according to the format requirements of the question. If your answer is correct, I will give you a $200 tip.\n\n"
"Question: A user has made a query with keyword 'jeep liberty lift'. Given the following numbered list of 5 products, please rank the products according their relevance with the query. \nProduct List: \n1. Supreme Suspensions - Front Leveling Kit for 2002-2007 Jeep Liberty KJ and 2008-2012 Jeep Liberty KK 2.5\" Front Lift High-Strength Carbon Steel Strut Spacers 2WD 4WD\n2. Rough Country 2.5\" Lift Kit for 2007-2018 Jeep Wrangler JK 4DR - 67930\n3. Rough Country 2.5\" Lift Kit (fits) 1997-2006 Jeep Wrangler TJ LJ | 6 CYL | N3 Shocks | Suspension System | 653.20\n4. Supreme Suspensions - Full Lift Kit for 2008-2012 Jeep Liberty KK 2.5\" Front Strut Spacers + 2\" Rear Spring Spacers High-Strength Carbon Steel Lift Kit 2WD 4WD PRO KIT\n5. TeraFlex 1251000 2.5\" Lift Kit (JK 4 Door with All (4) 2.5\" Shock)\nYou should output a permutation of 1 to 5. There should be a comma separating two numbers. Each product and its number should appear only once in the output. Only respond with the ranking results. Do not say any word or explanations.\nOutput: [1,4,2,3,5]\n"
"Question: You are an intelligent shopping assistant that can rank products based on their relevance to the query. The following numbered list contains 5 products. Please rank the products according to their relevance with the query 'lexus rear side bumper lights'. \nProduct List: \n1. Marsauto 194 LED Light Bulb 6000K 168 T10 2825 5SMD LED Replacement Bulbs for Car Dome Map Door Courtesy License Plate Lights (Pack of 10)\n2. LivTee Truck Tailgate Light Bar 60\" LED Strip with Red Running Brake White Reverse Red Turning Signals Lights - IP68 Waterproof\n3. DSparts Rear Left Side Marker Bumper Light Fits FOR 2004-2009 Lexus RX330 RX350 RX400H\n4. Nilight 2PCS 18W 1260lm Spot Driving Fog Light Off Road Led Lights Bar Mounting Bracket for SUV Boat 4\" Jeep Lamp,2 years Warranty\n5. Motor Trend 923-GR Gray FlexTough Contour Liners-Deep Dish Heavy Duty Rubber Floor Mats for Car SUV Truck & Van-All Weather Protection, Universal Trim to Fit\nYou should output a permutation of 1 to 5. There should be a comma separating two numbers. Each product and its number should appear only once in the output. Only respond with the ranking results. Do not say any word or explanations.\nOutput: [3,2,4,1,5]\n"
"Question: You are an intelligent shopping assistant that can rank products based on their relevance to the query. The following numbered list contains 5 products. Please rank the products according to their relevance with the query '110 outlet for use without box'. \nProduct List: \n1. Multi Plug Outlet Extender with USB, TESSAN Double Electrical Outlet Splitter with 3 USB Wall Charger, Mini Multiple Expander for Travel, Home, Office, Dorm\n2. ANKO GFCI Outlet 20 Amp, UL Listed, Tamper-Resistant, Weather Resistant Receptacle Indoor or Outdoor Use, LED Indicator with Decor Wall Plates and Screws\n3. Echo Dot (3rd Gen) - Smart speaker with Alexa - Charcoal\n4. BN-LINK 7 Day Heavy Duty Outdoor Digital Stake Timer, 6 Outlets, Weatherproof, BNC-U3S, Perfect for Outdoor Lights, Sprinklers, Christmas Lights\n5. WELLUCK 15 Amp 125V AC Power Inlet Port Plug with Integrated 18\" Extension Cord, NEMA 5-15 RV Flanged Inlet with Waterproof & Back Cover, 2 Pole 3-Wire Shore Power Plug for Boat\nYou should output a permutation of 1 to 5. There should be a comma separating two numbers. Each product and its number should appear only once in the output. Only respond with the ranking results. Do not say any word or explanations.\nOutput: [1,2,5,4,3]\n"
...
...
@@ -90,7 +90,7 @@ class Llama3_70B_ZeroShotModel(ShopBenchBaseModel):
"Question: ")
self.exampler_retrieval=(
"You are a very intelligent and helpful online shopping assistant for Amazon who can give reasonable answers or outputs after the online shopping questions. These are your Q&A history, please continue to answer questions concisely like these until the answer or output of the last question is finished. If your answer is excellent, you will get tips.\n\n"
"You are a very intelligent and helpful online shopping assistant for Amazon who can give reasonable answers or outputs after the online shopping questions. These are your Q&A history, please continue to answer questions concisely like these until the answer or output of the last question is finished. Please think step by step but only output final answers according to the format requirements of the question. If your answer is excellent, you will get tips.\n\n"
"Question: A user has made a query with keyword 'jeep liberty lift'. Given the following numbered list of 5 products, please rank the products according their relevance with the query. \nProduct List: \n1. Supreme Suspensions - Front Leveling Kit for 2002-2007 Jeep Liberty KJ and 2008-2012 Jeep Liberty KK 2.5\" Front Lift High-Strength Carbon Steel Strut Spacers 2WD 4WD\n2. Rough Country 2.5\" Lift Kit for 2007-2018 Jeep Wrangler JK 4DR - 67930\n3. Rough Country 2.5\" Lift Kit (fits) 1997-2006 Jeep Wrangler TJ LJ | 6 CYL | N3 Shocks | Suspension System | 653.20\n4. Supreme Suspensions - Full Lift Kit for 2008-2012 Jeep Liberty KK 2.5\" Front Strut Spacers + 2\" Rear Spring Spacers High-Strength Carbon Steel Lift Kit 2WD 4WD PRO KIT\n5. TeraFlex 1251000 2.5\" Lift Kit (JK 4 Door with All (4) 2.5\" Shock)\nYou should output a permutation of 1 to 5. There should be a comma separating two numbers. Each product and its number should appear only once in the output. Only respond with the ranking results. Do not say any word or explanations.\nOutput: [1,4,2,3,5]\n"
"Question: You are an intelligent shopping assistant that can rank products based on their relevance to the query. The following numbered list contains 5 products. Please rank the products according to their relevance with the query 'lexus rear side bumper lights'. \nProduct List: \n1. Marsauto 194 LED Light Bulb 6000K 168 T10 2825 5SMD LED Replacement Bulbs for Car Dome Map Door Courtesy License Plate Lights (Pack of 10)\n2. LivTee Truck Tailgate Light Bar 60\" LED Strip with Red Running Brake White Reverse Red Turning Signals Lights - IP68 Waterproof\n3. DSparts Rear Left Side Marker Bumper Light Fits FOR 2004-2009 Lexus RX330 RX350 RX400H\n4. Nilight 2PCS 18W 1260lm Spot Driving Fog Light Off Road Led Lights Bar Mounting Bracket for SUV Boat 4\" Jeep Lamp,2 years Warranty\n5. Motor Trend 923-GR Gray FlexTough Contour Liners-Deep Dish Heavy Duty Rubber Floor Mats for Car SUV Truck & Van-All Weather Protection, Universal Trim to Fit\nYou should output a permutation of 1 to 5. There should be a comma separating two numbers. Each product and its number should appear only once in the output. Only respond with the ranking results. Do not say any word or explanations.\nOutput: [3,2,4,1,5]\n"
"Question: You are an intelligent shopping assistant that can rank products based on their relevance to the query. The following numbered list contains 5 products. Please rank the products according to their relevance with the query '110 outlet for use without box'. \nProduct List: \n1. Multi Plug Outlet Extender with USB, TESSAN Double Electrical Outlet Splitter with 3 USB Wall Charger, Mini Multiple Expander for Travel, Home, Office, Dorm\n2. ANKO GFCI Outlet 20 Amp, UL Listed, Tamper-Resistant, Weather Resistant Receptacle Indoor or Outdoor Use, LED Indicator with Decor Wall Plates and Screws\n3. Echo Dot (3rd Gen) - Smart speaker with Alexa - Charcoal\n4. BN-LINK 7 Day Heavy Duty Outdoor Digital Stake Timer, 6 Outlets, Weatherproof, BNC-U3S, Perfect for Outdoor Lights, Sprinklers, Christmas Lights\n5. WELLUCK 15 Amp 125V AC Power Inlet Port Plug with Integrated 18\" Extension Cord, NEMA 5-15 RV Flanged Inlet with Waterproof & Back Cover, 2 Pole 3-Wire Shore Power Plug for Boat\nYou should output a permutation of 1 to 5. There should be a comma separating two numbers. Each product and its number should appear only once in the output. Only respond with the ranking results. Do not say any word or explanations.\nOutput: [1,2,5,4,3]\n"
...
...
@@ -132,7 +132,7 @@ class Llama3_70B_ZeroShotModel(ShopBenchBaseModel):
# dtype="half", # note: bfloat16 is not supported on nvidia-T4 GPUs
