Skip to content
Snippets Groups Projects
Commit 4e2c3116 authored by spmohanty's avatar spmohanty
Browse files

add generation task response to docs

parent 2bf03313
No related branches found
No related tags found
No related merge requests found
......@@ -21,6 +21,7 @@ The output from your model's `predict` function should always be a string. Depen
- A single integer (in the range [0, 3]) for multiple choice tasks.
- A comma-separated list of integers for ranking tasks.
- A comma-separated list of named entities for Named Entity Recognition (NER) tasks.
- (unconstrained) generated response for the generation tasks
For more information on how these responses are processed, please see [parsers.py](../parsers.py).
......
......@@ -18,6 +18,7 @@ class ShopBenchBaseModel:
str: The prediction as a string representing a single integer[0, 3] for multiple choice tasks,
or a string representing a comma separated list of integers for Ranking, Retrieval tasks,
or a string representing a comma separated list of named entities for Named Entity Recognition tasks.
or a string representing the (unconstrained) generated response for the generation tasks
Please refer to parsers.py for more details on how these responses will be parsed by the evaluator.
"""
raise NotImplementedError("predict method not implemented")
......@@ -35,6 +35,7 @@ class DummyModel(ShopBenchBaseModel):
str: The prediction as a string representing a single integer[0, 3] for multiple choice tasks,
or a string representing a comma separated list of integers for Ranking, Retrieval tasks,
or a string representing a comma separated list of named entities for Named Entity Recognition tasks.
or a string representing the (unconstrained) generated response for the generation tasks
Please refer to parsers.py for more details on how these responses will be parsed by the evaluator.
"""
possible_responses = [1, 2, 3, 4]
......@@ -46,3 +47,5 @@ class DummyModel(ShopBenchBaseModel):
# For other tasks, shuffle the possible responses and return as a string
random.shuffle(possible_responses)
return str(possible_responses)
# Note: As this is dummy model, we are returning random responses for non-multiple choice tasks.
# For generation tasks, this should ideally return an unconstrained string.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment