The vocab.txt links in tokenization_unilm.py are deprecated. Need to update the latest checkpoints links according to [link].(https://unilm.blob.core.windows.net/ckpt/unilm1.2-base-uncased-vocab.txt)

PRETRAINED_VOCAB_FILES_MAP = {
    'vocab_file':
    {
        'unilm-large-cased': "https://conversationhub.blob.core.windows.net/beit-share-public/ckpt/unilm-large-cased-vocab.txt",
        'unilm-base-cased': "https://conversationhub.blob.core.windows.net/beit-share-public/ckpt/unilm-base-cased-vocab.txt",
        'unilm1-large-cased': "https://conversationhub.blob.core.windows.net/beit-share-public/ckpt/unilm1-large-cased-vocab.txt",
        'unilm1-base-cased': "https://conversationhub.blob.core.windows.net/beit-share-public/ckpt/unilm1-base-cased-vocab.txt",
        'unilm1.2-base-uncased': "https://conversationhub.blob.core.windows.net/beit-share-public/ckpt/unilm1.2-base-uncased-vocab.txt"
    }
}

Thus, links in configuration_minilm.py, configuration_unilm.py, tokenization_minilm.py, tokenization_unilm.py need to be updated as well.

Hello, I'm trying to generate summaries on dialogues using the pre-trained model (MingZhong/DialogLED-large-5120) as well as a fine-tuned one (on AMI dataset). It does not produce any meaningful summaries, but instead it just replicates a part of the dialogue. I'm using an adaptation of AllenAI (allenai/led-large-16384-arxiv) and the Huggingface LEDTokenizer and LEDForConditionalGeneration scripts:

LONG_DIALOGUE = """ Here goes my long dialogue."""" import torch from transformers import LEDTokenizer, LEDForConditionalGeneration

tokenizer = LEDTokenizer.from_pretrained("AMI_DialogLED_large/") input_ids = tokenizer(LONG_DIALOGUE, return_tensors="pt").input_ids.to("cuda") global_attention_mask = torch.zeros_like(input_ids) global_attention_mask[:, 0] = 1

model = LEDForConditionalGeneration.from_pretrained("AMI_DialogLED_large/", return_dict_in_generate=True).to("cuda") sequences = model.generate(input_ids, global_attention_mask=global_attention_mask).sequences

summary = tokenizer.batch_decode(sequences) print(summary)

With this code the output is a gibberish dialogue, not an actual summary as shown in your model outputs. Is there another way to generate summaries, or should the model be used in a different way?

Thanks,

David

Hi! Thank you for sharing the dialogLM and dialogLED implementation. I wonder if it's possible to release the hyperparameters used for ForeverDreaming and TVMegaSite. I see the DialogLM_UniLM folder does use TVMegaSite as an example and provided the hyperparameters. Does fine-tuning on ForeverDreaming use slightly different parameters? In particular, knowing the num_training_steps and batchsize will be super useful for us to replicate the results presented in the paper.

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 1199) of binary: /usr/bin/python3 Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.7/dist-packages/torch/distributed/launch.py", line 193, in main() File "/usr/local/lib/python3.7/dist-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/usr/local/lib/python3.7/dist-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/usr/local/lib/python3.7/dist-packages/torch/distributed/run.py", line 755, in run )(*cmd_args) File "/usr/local/lib/python3.7/dist-packages/torch/distributed/launcher/api.py", line 131, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/usr/local/lib/python3.7/dist-packages/torch/distributed/launcher/api.py", line 247, in launch_agent failures=result.failures, torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

As mentioned in the research paper -

DIALOGLM is obtained by further pre-training UNILMbase with the window-based denoising method. Its maximum input length is 5,120 and the tokens exceeding this length is truncated in the experiments. DIALOGLM-sparse additionally introduces the hybrid attention approach in the pre-training process of DIALOGLM, so its maximum length is increased to 8,192 tokens.

I am not understanding how DialogLM is able process 5,120 tokens without using hybrid attention approach? Since it is using UniLM V2 as backbone the max tokens it can process should be 512 right?