I have trained a baseline model and run prediction on validation split according to the instructions in the baseline README. However, the command line output didn't seem to give me a destination folder that contains the generated predictions after running through the validation dataset. I was hoping to find a JSON file containing the validation split predictions so that I can use that in the evaluator. Is there a way that I can find the validation split predictions?

Moreover, is there a way for me to evaluate the results on the test split? I see the README in the evaluator folder which has options Evaluate predictions for a single dataset (validation only) Evaluate predictions for the entire benchmark (validation only) Prepare Submission File Verify Submission File
I want to evaluate the metrics on the test dataset (to see if the resulting numbers match the paper), but I don't want to generate a submission file since I'm just running the baseline models. Is there a way to do that? Thank you very much!

EDIT: I'm currently only running the QMSUM dataset, not the others.

For the your tasks, are the "input" column the full input passed into the models? Did you add any additional prompting for the models provided in the leaderboard?

For example, for GovReport, did you (or were the teams who submitted allowed to) do something like the following?

Original Text:
<"input" column of GovReport>

Summary:
<"output" column of GovReport / output of model>

Or is there no additional prompting:

<"input" column of GovReport>
<"output" column of GovReport / output of model>

1. From the paper, you seem to truncate inputs to 16,384 tokens for your leaderboard, is that right?
2. As n-gram metrics are affected by the length of outputs, how do you determine the target length of outputs? I notice that the default max_target_length in baselines/src/run.py is 128 tokens. Do you train your models with an EOS token such that the generated output may terminate much earlier?

First I want to thank the authors for this great work! I might find it useful for my research.

I encountered 3 problems:

1. in evaluator/dataset_evaluator.py, in the usage of hf_hub_download, I got an exception because it was used that way: hf_hub_download(repo_id="datasets/tau/scrolls", filename="metrics/scrolls.py") instead of hf_hub_download(repo_id="tau/scrolls", filename="metrics/scrolls.py", repo_type="dataset"). I don't know why it worked for you, perhaps there was a breaking change in the datasets library recently. Would you want me to open a PR for that?
2. The generate script (python scripts/execute.py scripts/commands/generate.py {dataset}_{model}_{split} --checkpoint_path path/to/model/folder) took a very long time, much more than the fine-tuning of 256-bart. There was a warning that might be related saying:

Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector

Edit: I now noticed that this warning is emitted only when I use more than one GPU. However, it is still slower than expected. 3. It failed with the following exception:

Traceback (most recent call last): File "/home/liranringel/scrolls/baselines/scripts/execute.py", line 53, in main(command_dict, unknown) File "/home/liranringel/scrolls/baselines/scripts/execute.py", line 33, in main runpy.run_module(module_name, run_name="main") File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/runpy.py", line 228, in run_module return _run_code(code, {}, init_globals, run_name, mod_spec) File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/liranringel/scrolls/baselines/src/run.py", line 789, in main() File "/home/liranringel/scrolls/baselines/src/run.py", line 656, in main metrics = trainer.evaluate(metric_key_prefix="eval") File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/site-packages/transformers/trainer_seq2seq.py", line 131, in evaluate eval_preds = self._post_process_function(untokenized_eval_dataset, eval_loop_output.predictions) File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/site-packages/transformers/trainer_seq2seq.py", line 326, in _post_process_function assert len(untokenized_eval_dataset) == len(self.eval_dataset) AssertionError