Hi, thanks for your great work. I wander how to generate the prediction results file. For example, there is no related scripts in UBAR code, how to generate the predicted results in the expected format?

Hi,

I am investigating about pipeline task-oriented agents by using the ConvLab-2 toolkit and I have some doubts regarding its results. I think that their multiwoz evaluation method is not the same, although they use same metrics. Have you ever evaluate any ConvLab-2 agent by using your scripts?

Ref: https://github.com/thu-coai/ConvLab-2

Following the question by @SkyAndCloud in #1 and our email communication:

In the new evaluation, you noted in README.md that the inputs are: 1) generated response 2) generated belief states 3) active domains ... It does not need the generated system actions or generated db to calculate the metrics. My question is that, in the old evaluation scripts (I mean the original implementation inside the multiwoz, https://github.com/budzianowski/multiwoz/blob/master/evaluate.py), are the inputs these three (generated response, generated belief states, active domains) too? Does the old evaluation need the generated system actions or generated db to calculate the metrics ?

Hello, We've used your standardized scripts to (re-)evaluate our RSTOD architecture on MultiWOZ 2.2. We're also opening a pull request in the official dataset repository to report benchmark results.

Please check it out :) Best Regards, Radostin

Hi,

Our GALAXY model uses your standardized scoring scripts to conduct new evaluations on both end-to-end modeling and policy optimization tasks. We have already reported the new SOTA results in the official MultiWOZ repo. So we upload two prediction files here, where 'galaxy-e2e.json' for the end-to-end generation while 'galaxy-policy.json' for the policy optimization.

Please check it. Thanks a lot!

Best, Wanwei

# Go through the venues returned by the API call matching the dialog state of the current turn and # use them as the list of possibly offered venues if and only if any of them is not present in the # list of possibly offered venues from the previous dialog turn if current_domain not in offered_venues or len(offered_venues[current_domain]) == 0: offered_venues[current_domain] = matching_venues else: if any(venue not in matching_venues for venue in offered_venues[current_domain]): ^ according to the comment above, is this line supposed to be any(venue not in offered_venues[current_domain] for venue in matching_venues) offered_venues[current_domain] = matching_venues

Hello Guys,

Just upload the prediction file from our new paper PPTOD (https://arxiv.org/abs/2109.14739). We also updated our scores in the main repo (https://github.com/budzianowski/multiwoz).

Thank you very much!

Best,

Yixuan

Hello,

We upload the prediction files from our papers Mars (https://arxiv.org/abs/2210.08917) and BORT (https://aclanthology.org/2022.findings-naacl.166/). We also updated our scores in the main repo (https://github.com/budzianowski/multiwoz).

Thank you very much!

Best,

Haipeng

This PR adds the possibility of evaluation dialog state tracking on MultiWOZ 2.2 test set.

The implementation involves slot value normalization and careful fuzzy matching, so it produces much higher accuracy than other implementations.

It calculates:

• joint accuracy -- the percentage of turns that have exactly the same dialog state as the reference
• slot F1, precision, recall, where:
  • TP are slots mentioned in the reference and prediction with matching value
  • TN are not calculated at all, but it should correspond to slots that are not mentioned in the reference nor prediction
  • FN are slots that are not mentioned in the prediction but are in the reference
  • FP are those slots mentioned in the prediction but not mentioned in the reference or mentioned in both but not matching by value

I am testing the UBAR method in mwoz2.2 and created a parser to convert back the generated sequence into a state dict. First I loaded groundtruth data with datasets and parsed to "predicted" dictionary, then I computed the metrics:

datasets = load_dataset("json", data_files={
            "train": "data/multiwoz/train/encoded.json",
            "valid": "data/multiwoz/dev/encoded.json",
            "test": "data/multiwoz/test/encoded.json",
        })

predicted = {}

for d in datasets["test"]:
    id = d["id"].rstrip(".json").lower()
    turns = []
    for belief in d["text"].split("<sos_b>")[1:]:
        bs = parse_state(belief.split("<eos_b>")[0])
        response = belief.split("<sos_r>")[1].split("<eos_r>")[0]
        state = {"response": response, "state":{}}
        for k,v in bs:
            state["state"][k] = v
        turns.append(state)
    predicted[id] = turns

e = Evaluator(bleu=True, success=True, richness=True)
results = e.evaluate(predicted)
print(results)

>>> {'bleu': {'mwz22': 99.15078876656015},
'success': {'inform': {'total': 93.0, 'train': 95.6, 'restaurant': 95.9, 'hotel': 95.9, 'taxi': 100.0, 'attraction': 96.0}, 'success': {'total': 88.1, 'train': 89.1, 'restaurant': 90.2, 'hotel': 87.6, 'taxi': 90.8, 'attraction': 90.7}},
'richness': {'entropy': 7.218217822046144, 'cond_entropy': 3.3791865228994378, 'avg_lengths': 14.094411285946826, 'msttr': 0.7501539942252144, 'num_unigrams': 1467, 'num_bigrams': 11614, 'num_trigrams': 25497}, 'dst': None}

I didn't get the 100% for inform and success.

Then I compared it with the golden states:

with open("venv/lib/python3.10/site-packages/mwzeval/data/gold_states.json") as fin:
        data = json.load(fin)

counter = 0
for key in predicted:
    for i, value in enumerate(predicted[key]):
        if value["state"] != data[key][i]:
            counter += 1
            print(key, i)
            pprint(value["response"])
            pprint(value["state"])
            pprint(data[key][i])
if not counter:
    print("100% matched")

As output I got that it matched 100%. It is the expected behavior? I am missing something?