I have a question about json data in public.

1. What do each of the four values in bbox represent?
   • How do I change the bbox+position to (left, top, right, bottom)?
2. Does relationships represent a relationship between indices?
3. What is the difference between unique_id and index?

Thank you.

@satwikkottur I add minor corrects to the code. Please let me know if they are incorrect

1. Delete tokenizer.padding_side = "left". It seems that after removing this line, the accuracy could be improved from 72% to 92% in the devtest set.
2. Add tokenizer.truncation_side = "left" to enable truncate oldest context tokens when needed.

Hi, you have released the simmc2_dials_dstc10_teststd_retrieval_candidates_public.json data for the challenge phase 2. But I found out that the original the teststd data contains only one turn of candidates in each dialog. Since the original simmc2_dials_dstc10_devtest_retrieval_candidates.json file had candidates of multiple turns in a dialog, I just wanted to make sure that this modification was made on purpose. Thank you!

Hello.

When I solve subtask1, I have a question.

subtask2 needs to find an object reference corresponding to the current utterance.

Is this multimodal information available in subtask1? That is, when predicting disambiguation, is it okay to use multimodal information (e.g. object bounding bbox and metatdata) corresponding to the current utterance? Or can I just use information about the whole image corresponding to the dialog?

Thank you.

Hi,

In evaluate_response.py, I see the following snippet

def parse_response_from_file(input_path): """Parses the response from a flattened file. Args: input_path: Path to read the responses from. """ lines = [] with open(input_path, "r") as file_id: for ii in file_id.readlines(): split_line = ii.split("<SOR>", 1) lines.append( (split_line[0].strip("\n"), split_line[1].strip("\n").strip("")) ) return lines

Here we have <SOR>, but this is only used at noblief mode, while the baseline also uses belief. Is it allowed to fix evaluation code a little for cases like this? or should I conform to this eval script?

Hi,

I am facing the issue that some object IDs do not appear in the scene file for that dialogue. For example, in m_cloth_store_1498649_woman_5_9, there is a reference to object ID 55, but there is no object ID 55 either as index or unique_id. Could you please clarify this?

This question was also asked by @tungngthanh, (quoted below), as the issue was closed without an answer.

After a quick count, this happens ~615 times in the train split: the target object ID does not appear in the scene or bbox jsons for that dialogue.

Thank you.

EDIT: if I include the objects in system_transcript_annotated and transcript_annotated together, ~3183 entries (user utterance + dialogue history up to that point) make references to objects that do not appear in the respective scene jsons, around 8% of the train data. I am skipping these for now.

Thank you for your reply. Follow your suggestions, I find that each scene idx corresponds to one json file and one image file. However, now I face some problems when I mapped the object_local_id to the canonical object_id. For example, the third dialogue (index 2) in the training set which has the scene_ids as follows: {'0': 'm_cloth_store_1498649_woman_5_3', '5': 'm_cloth_store_1498649_woman_5_9'} its 7th uterance is : { 'turn_idx': 6, 'system_transcript': "Sure, I'll add that now.", 'system_transcript_annotated': {'act': 'CONFIRM:ADD_TO_CART', 'act_attributes': {'slot_values': {}, 'request_slots': [], 'objects': [55]}}, 'transcript': 'Actually, just add that brown jacket to my cart.', 'transcript_annotated': {'act': 'REQUEST:ADD_TO_CART', 'act_attributes': {'slot_values': {}, 'request_slots': [], 'objects': [55]}} } According to the document, we should expect to see the local_id 55 object in m_cloth_store_1498649_woman_5_9_scene.json, right? However, when I load the file, I do not see 55 in index or unique_id. Can you clarify it for me?

Originally posted by @tungngthanh in https://github.com/facebookresearch/simmc2/issues/3#issuecomment-896868063

At simmc2_dials_dstc10_devtest_target.txt, the following line exits:

User : I'm interested in a hoodie. System : How do you feel about this brown one here on the front floor rack, and the brown one in front of it? They are both hoodies. 47, 50 User : What's the prive of the item? => Belief State : ASK:GET [ ] (price) < 36 > Which item do you mean?

But I think it's impossible to correctly predict the object at Belief State which is 36, when the model is given only the part before Belief State, because the user utterance contains ambiguity in it (so the system actually disambiguates it!).

If I'm right, this is a problem. Please let me know.

There is an error on parse_flattened_result function in model/mm_dst/gpt2_dst/utils/convert.py

This function returns empty array when it handles strings contain nested square bracktes.

e.g.) ".. INFORM:GET [ sleeveLength = short, availableSizes = ['XXL', 'S', 'L'], pattern = leafy design, type = blouse ] (availableSizes, pattern) < 86, 57 > ... "

Hi,

I think some of the links in the README.md are broken or files are missing. For instance, the following:

Please check the [task input](./TASK_INPUTS.md) file for a full description of inputs for each subtask.

It references TASK_INPUTS.md but I cannot find the file anywhere in the repository and opening it takes me to a 404 Not Found webpage. Are there files missing from the repo by any chance? Thanks! :)

Hello. I am a DSTC10 participant.

Thanks for the data release.

After downloading and checking the data, it is currently stored as follows in simmc2_dials_dstc10_train.json.

It's different from what you described in the data introduction. Is there an update?

Thank you.

"disambiguation candidate raw" seems to narrow down the object subspace. Normally, discrete indexes are listed, but there are times where the values are "all", "blouse", "dress" etc. What does these string type values mean?

I guessed that "all" and "blouse" would mean all blouses in the given scene. I'm also curious if "disambiguation candidate raw" could be used at testing.

​Hello, I would like to raise a question regarding SIMMC 2.1 dataset provided here.​

During creating my own data preprocessing script, I noticed that some of the coreference labels (obtained from the dialogue JSON data under transcript_annotated > act_attributes > objects; following MM-DST's preprocessing script) are not a subset of the corresponding object maps used by the relevant dialogue id and turn id. I extracted these object maps following this function from the preprocessing script of ambiguous candidate identification. For clarity, here are the mismatched label/target set and object map pairs from the devtest data.

dialog_id 10618 | turn_id 7 | image_name cloth_store_1416238_woman_3_9.png | scene_label m_cloth_store_1416238_woman_3_9 | target {57, 2} | object_map {85, 86, 87, 56, 57, 58, 59, 61, 62, 63}
dialog_id 10653 | turn_id 8 | image_name cloth_store_1416238_woman_4_6.png | scene_label m_cloth_store_1416238_woman_4_6 | target {2, 59} | object_map {1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 76, 77, 78, 79, 80, 81, 82, 83}
dialog_id 10677 | turn_id 7 | image_name cloth_store_1498649_woman_20_3.png | scene_label m_cloth_store_1498649_woman_20_3 | target {52, 53} | object_map {19, 20, 21, 22, 24, 25, 26, 27, 31, 32, 33, 34, 35, 36, 37, 44, 45, 46, 47}
dialog_id 10677 | turn_id 8 | image_name cloth_store_1498649_woman_20_3.png | scene_label m_cloth_store_1498649_woman_20_3 | target {52, 53} | object_map {19, 20, 21, 22, 24, 25, 26, 27, 31, 32, 33, 34, 35, 36, 37, 44, 45, 46, 47}
dialog_id 10743 | turn_id 8 | image_name cloth_store_1498649_woman_20_10.png | scene_label m_cloth_store_1498649_woman_20_10 | target {21} | object_map {0, 1, 2, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 23, 28, 29, 38, 40, 43, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66}
dialog_id 10743 | turn_id 9 | image_name cloth_store_1498649_woman_20_10.png | scene_label m_cloth_store_1498649_woman_20_10 | target {21} | object_map {0, 1, 2, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 23, 28, 29, 38, 40, 43, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66}
dialog_id 10769 | turn_id 8 | image_name cloth_store_1498649_woman_20_9.png | scene_label m_cloth_store_1498649_woman_20_9 | target {15} | object_map {0, 1, 2, 3, 4, 5}
dialog_id 10788 | turn_id 7 | image_name cloth_store_1498649_woman_20_6.png | scene_label m_cloth_store_1498649_woman_20_6 | target {71} | object_map {0, 1, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43}
dialog_id 10873 | turn_id 7 | image_name cloth_store_1498649_woman_20_5.png | scene_label m_cloth_store_1498649_woman_20_5 | target {9, 14} | object_map {48, 51, 52, 53, 54, 59, 60, 61, 62, 63, 64, 67, 68, 69, 70, 71, 72, 73, 74, 75}
dialog_id 10873 | turn_id 8 | image_name cloth_store_1498649_woman_20_5.png | scene_label m_cloth_store_1498649_woman_20_5 | target {9} | object_map {48, 51, 52, 53, 54, 59, 60, 61, 62, 63, 64, 67, 68, 69, 70, 71, 72, 73, 74, 75}
dialog_id 10896 | turn_id 8 | image_name cloth_store_1498649_woman_2_9.png | scene_label m_cloth_store_1498649_woman_2_9 | target {8, 18} | object_map {0, 1, 2, 40, 42, 44, 45, 14, 15, 46, 19, 20, 53, 21, 23, 24, 60}
dialog_id 10942 | turn_id 7 | image_name cloth_store_1416238_woman_3_11.png | scene_label m_cloth_store_1416238_woman_3_11 | target {40, 3} | object_map {39, 40, 41, 42, 43, 44, 45}
dialog_id 10966 | turn_id 8 | image_name cloth_store_1498649_woman_20_5.png | scene_label m_cloth_store_1498649_woman_20_5 | target {7} | object_map {48, 51, 52, 53, 54, 59, 60, 61, 62, 63, 64, 67, 68, 69, 70, 71, 72, 73, 74, 75}
dialog_id 10981 | turn_id 9 | image_name cloth_store_1416238_woman_4_5.png | scene_label m_cloth_store_1416238_woman_4_5 | target {33, 9} | object_map {15, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36}
dialog_id 11030 | turn_id 9 | image_name cloth_store_1498649_woman_20_1.png | scene_label m_cloth_store_1498649_woman_20_1 | target {33, 36} | object_map {0, 1, 2, 3, 4, 5}
dialog_id 11092 | turn_id 6 | image_name cloth_store_1416238_woman_4_5.png | scene_label m_cloth_store_1416238_woman_4_5 | target {60, 37} | object_map {15, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36}
dialog_id 11107 | turn_id 7 | image_name cloth_store_1498649_woman_2_9.png | scene_label m_cloth_store_1498649_woman_2_9 | target {34, 39} | object_map {0, 1, 2, 40, 42, 44, 45, 14, 15, 46, 19, 20, 53, 21, 23, 24, 60}
dialog_id 11150 | turn_id 9 | image_name cloth_store_1498649_woman_20_2.png | scene_label m_cloth_store_1498649_woman_20_2 | target {17, 18} | object_map {48, 51, 52, 53, 54, 55, 59, 60, 61, 62, 63, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78}
dialog_id 11179 | turn_id 9 | image_name cloth_store_1498649_woman_20_10.png | scene_label m_cloth_store_1498649_woman_20_10 | target {69} | object_map {0, 1, 2, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 23, 28, 29, 38, 40, 43, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66}
dialog_id 11258 | turn_id 6 | image_name cloth_store_1416238_woman_4_3.png | scene_label m_cloth_store_1416238_woman_4_3 | target {63, 71} | object_map {97, 98, 99, 100, 84, 85, 86, 87, 88, 89, 90, 91, 92}
dialog_id 11325 | turn_id 8 | image_name cloth_store_1416238_woman_3_11.png | scene_label m_cloth_store_1416238_woman_3_11 | target {25} | object_map {39, 40, 41, 42, 43, 44, 45}
dialog_id 11509 | turn_id 7 | image_name cloth_store_1498649_woman_2_1.png | scene_label m_cloth_store_1498649_woman_2_1 | target {1, 45} | object_map {0, 1, 41, 43, 44, 14, 15, 51, 52, 53, 20, 23, 21, 24, 59, 61, 62, 63}
dialog_id 11584 | turn_id 7 | image_name cloth_store_1416238_woman_3_9.png | scene_label m_cloth_store_1416238_woman_3_9 | target {3, 6} | object_map {85, 86, 87, 56, 57, 58, 59, 61, 62, 63}
dialog_id 11630 | turn_id 9 | image_name cloth_store_1416238_woman_4_10.png | scene_label m_cloth_store_1416238_woman_4_10 | target {0, 6} | object_map {0, 9, 10, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50}
dialog_id 11677 | turn_id 8 | image_name cloth_store_1416238_woman_4_3.png | scene_label m_cloth_store_1416238_woman_4_3 | target {59} | object_map {97, 98, 99, 100, 84, 85, 86, 87, 88, 89, 90, 91, 92}
dialog_id 11677 | turn_id 9 | image_name cloth_store_1416238_woman_4_3.png | scene_label m_cloth_store_1416238_woman_4_3 | target {59} | object_map {97, 98, 99, 100, 84, 85, 86, 87, 88, 89, 90, 91, 92}
dialog_id 11738 | turn_id 7 | image_name cloth_store_1416238_woman_3_9.png | scene_label m_cloth_store_1416238_woman_3_9 | target {72, 49} | object_map {85, 86, 87, 56, 57, 58, 59, 61, 62, 63}
dialog_id 11738 | turn_id 8 | image_name cloth_store_1416238_woman_3_9.png | scene_label m_cloth_store_1416238_woman_3_9 | target {49} | object_map {85, 86, 87, 56, 57, 58, 59, 61, 62, 63}
dialog_id 11792 | turn_id 8 | image_name cloth_store_1416238_woman_3_9.png | scene_label m_cloth_store_1416238_woman_3_9 | target {8} | object_map {85, 86, 87, 56, 57, 58, 59, 61, 62, 63}
dialog_id 11792 | turn_id 9 | image_name cloth_store_1416238_woman_3_9.png | scene_label m_cloth_store_1416238_woman_3_9 | target {8} | object_map {85, 86, 87, 56, 57, 58, 59, 61, 62, 63}
dialog_id 11839 | turn_id 6 | image_name cloth_store_1416238_woman_3_11.png | scene_label m_cloth_store_1416238_woman_3_11 | target {10, 46} | object_map {39, 40, 41, 42, 43, 44, 45}
dialog_id 11866 | turn_id 6 | image_name cloth_store_1498649_woman_20_2.png | scene_label m_cloth_store_1498649_woman_20_2 | target {57} | object_map {48, 51, 52, 53, 54, 55, 59, 60, 61, 62, 63, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78}
dialog_id 11869 | turn_id 8 | image_name cloth_store_1498649_woman_20_3.png | scene_label m_cloth_store_1498649_woman_20_3 | target {78, 55} | object_map {19, 20, 21, 22, 24, 25, 26, 27, 31, 32, 33, 34, 35, 36, 37, 44, 45, 46, 47}
dialog_id 12059 | turn_id 8 | image_name cloth_store_1498649_woman_20_5.png | scene_label m_cloth_store_1498649_woman_20_5 | target {6} | object_map {48, 51, 52, 53, 54, 59, 60, 61, 62, 63, 64, 67, 68, 69, 70, 71, 72, 73, 74, 75}
dialog_id 12128 | turn_id 6 | image_name cloth_store_1416238_woman_4_3.png | scene_label m_cloth_store_1416238_woman_4_3 | target {11, 12} | object_map {97, 98, 99, 100, 84, 85, 86, 87, 88, 89, 90, 91, 92}
dialog_id 12146 | turn_id 8 | image_name cloth_store_1416238_woman_3_1.png | scene_label m_cloth_store_1416238_woman_3_1 | target {56, 86} | object_map {60, 64, 69, 70, 71, 72, 73, 74, 75, 76, 81, 82, 83, 84, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100}
dialog_id 12167 | turn_id 9 | image_name cloth_store_1416238_woman_3_3.png | scene_label m_cloth_store_1416238_woman_3_3 | target {43} | object_map {32, 3, 4, 5, 6, 7, 8, 9, 10, 37, 76, 84, 29}
dialog_id 12170 | turn_id 6 | image_name cloth_store_1498649_woman_2_5.png | scene_label m_cloth_store_1498649_woman_2_5 | target {40, 2} | object_map {32, 33, 34, 35, 36, 37, 38, 39, 25, 26, 27, 28, 29, 30, 31}
dialog_id 12236 | turn_id 6 | image_name cloth_store_1416238_woman_4_0.png | scene_label m_cloth_store_1416238_woman_4_0 | target {98, 85} | object_map {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14}
dialog_id 12385 | turn_id 6 | image_name cloth_store_1498649_woman_20_9.png | scene_label m_cloth_store_1498649_woman_20_9 | target {60, 70} | object_map {0, 1, 2, 3, 4, 5}
dialog_id 12400 | turn_id 8 | image_name cloth_store_1498649_woman_20_10.png | scene_label m_cloth_store_1498649_woman_20_10 | target {34, 18} | object_map {0, 1, 2, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 23, 28, 29, 38, 40, 43, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66}
dialog_id 12479 | turn_id 8 | image_name cloth_store_1498649_woman_20_5.png | scene_label m_cloth_store_1498649_woman_20_5 | target {9, 11} | object_map {48, 51, 52, 53, 54, 59, 60, 61, 62, 63, 64, 67, 68, 69, 70, 71, 72, 73, 74, 75}
dialog_id 12482 | turn_id 8 | image_name cloth_store_1498649_woman_20_0.png | scene_label m_cloth_store_1498649_woman_20_0 | target {4, 6} | object_map {6, 7, 10, 13, 48, 49, 50, 51, 52, 53, 54}

I would like to know if these differences are intentional. If not, I wonder how the corresponding data instances should be handled during the evaluation process.

PS: If you can't reproduce these mismatched results, my preprocessing script could be incorrect. In that case, it would be great for me to know how the coreference labels or the object maps should be extracted to work as intended.

Hi，SIMMC Organizers!

When we analyze disambiguation annotations provided by SIMMC 2.1 Task1, we are a little confused about "disambiguation_candidates" field and "disambiguation_candidates_raw" field.

We find that sometimes "disambiguation_candidates_raw" field is "all items" even concrete description, like “light blue jeans”, has existed in the user utterance.

SIMMC 2.1 Dev dataset -- Dialogue ID 12216, Turn 3

Last User: Just go ahead and add the blue jeans to my cart.

Last System: Okay, they will be added.

Curr User: Now tell me the size and brand for the light blue jeans.

Curr System: Which ones?

Disam Raw: ['all', 'items']

Disam Candidates: [25, 26, 28, 29, 30, 31, 34, 35, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 61, 97, 98, 99]

We also find that "disambiguation_candidates_raw" field contains object id which are not related to the object type mentioned in the user utterance. The following dialogue disambiguation relates to "brown jacket" as user says but "disambiguation_candidates_raw" contains brown pants.

SIMMC 2.1 Dev dataset -- Dialogue ID 10858, Turn 7

Last User: Also, I need some large shoes.

Last System: Tell me what you think of the red shoes on the bottom of the right dresser?

Curr User: Actually, could you give me the available sizes of the brown jacket?

Curr System: Which one are you talking about?

Disam Raw: [60, 69, 81, 86, 92]

Disam Candidates: [60, 69, 81, 86, 92]

Besides, in some cases, there is only one object ID exists in "disambiguation_candidates_raw" field and "disambiguation_candidates" field, which seems contradictory to the goal of disambiguation task. For example,

SIMMC 2.1 Dev dataset -- Dialogue ID 11088, Turn 3

Last User: I need a solid color hoodie from 212 Local.

Last System: Here's a brown one on the far left, check it out.

Curr User: How much does that brown hoodie cost?

Curr System: Sorry, which one?

Disam Raw: [45]

Disam Candidates: [45]

Therefore, we want to ask for some help from you. Can you tell us the annotation process about SIMMC 2.1 Task1 or how to determine the scope of disambiguation candidates object IDs. Thank you very much!

Hello, I'm currently trying to implement a model for subtask 4b but I'm not able to find information about how long the list of answer candidates should be for the retrieval task. I would be really grateful for some clarification.

Thanks in advance and best regards Manuel

Maybe some mistakes, but I am not sure.

1. As far as I understand, '3.Generate prediction for devtest data' generated prediction 'simmc2.1_dials_dstc11_devtest_predicted.txt' for '4. Evaluate predictions for devtest data'. but the value of 'path_output' and 'input_path_predicted' are not consistent

2, In the results.

I am confusing the reported results are SIMMC 2 or 2.1

When I convert the flat text format into submission format, I see some inconsistencies in evaluation scripts. Can you clarify for me? Subtask 1 In ./model/utils/disambiguator_evaluation.py line 46:

assert "disambiguation_label" in gt_datum, "Turn not to be evaluated!"

This line will make the error when the groundtrue data do not have disambiguation_label. A lot of dialogue turn do not have the label so it causes the error when evaluating the dataset.

Subtask 3 In ./model/mm_dst/utils/evaluate_dst.py line [272] (https://github.com/facebookresearch/simmc2/blob/master/model/mm_dst/utils/evaluate_dst.py#L272)

true_frame_object_values == pred_frame_object_values

I think the n_correct_beliefs should not be related to frame_object_values, right?

Subtask 4 In ./model/utils/retrieval_evaluation.py According to the code, the expected format should be

 [
    "dialog_id": <dialog_id>,
    "candidate_scores": [
        {
            "turn_id": <turn_id>,
            "scores": [
                <list of 100 floats>
            ]
        }
        ...
    ]
    ...
]