I downloaded the pre-trained COMET with BART, and executed run.sh without --do-train and gave the path of the downloaded model to --model_name_or_path however, the test RougeL is not what you reported in the paper:

I downloaded the test files from [here] as mentioned in other issue (https://storage.googleapis.com/ai2-mosaic-public/projects/mosaic-kgs/data_atomic_2020_BART-format.tgz ) It's the content of metrics.json

            "test_avg_loss": 4.657896041870117,
            "test_avg_rouge1": 0.2141809276565792,
            "test_avg_rouge2": 0.03116943751406806,
            "test_avg_rougeL": 0.21284768699554912,
            "test_avg_gen_time": 0.003392871381747925,
            "test_avg_summ_len": 7.352367184676545,
            "avg_rouge1": 0.2141809276565792,
            "step_count": 1

Hi,

Thanks for putting together the repo! I'm trying to make predictions with the model using the models/comet_atomic2020_bart/generation_example.py script. But the behavior is sometimes a little worse than that of the AI2 demo.

Could you please specify the model architecture/hyperparameter settings that are used by the demo? Thanks in advance!

Hello sir.

I tried to run your codes that use BART model to generate knowledge triples.

In your codes, "models/comet_atomic2020_bart/finetune.py" requires "train.source" file and "train.target" file...

However, I couldn't figure out how to get these files.

How can I get these files?

Thanks.

There are duplicate tuples in all three splits of data: ~68,626 in the train, ~7,410 in dev, and ~8,473 in test (please correct me if I'm wrong). I wonder why? And should we just ignore the duplicates when using the data? One example:

['PersonX answers the question', 'xAttr', 'knowledgeable']
['PersonX answers the question', 'xAttr', 'knowledgeable']

Also, there are tuples with none tail node value (these none valued tuples are also part of the duplicate tuples). For example,

['PersonX accidentally threw ___', 'xIntent', 'none']
['PersonX accidentally threw ___', 'xIntent', 'none']

I wonder how these none values should be interpreted? Should we just ignore them? Or, does it mean that the subject or head has no relation of type edge relation in the tuple? For instance, in the case of PersonX accidentally threw ___, PersonX has no xIntent? If that's the case, then how should we treat the following cases:

['PersonX accidently left', 'oReact', 'none']
['PersonX accidently left', 'oReact', 'sad']

Where we have the same relation one time with a none tail node and another time with a non-empty tail node.

Thanks.

When I tried to run:

py eval.py --gen_file ../models/comet_atomic2020_gpt2/results/pred_generations.jsonl

I got the following error:

Traceback (most recent call last):
  File "eval.py", line 139, in <module>
    sources, references, predictions = preprocess(args.gen_file, keys)
  File "eval.py", line 109, in preprocess
    keys_list = keys if keys!=None else generations[0]["generations"].keys()
AttributeError: 'list' object has no attribute 'keys'

What are keys? the generated list has no keys,

You also pass keys to the evaluation, but what are them?

python anli_evaluation/eval.py --gen_file GENERATIONS_FILE --keys MODEL_KEYS[comma-separated list] --results_file RESULTS_FILE

I also checked automatic_eval.py in system_eval, it gets different types of input (type 1, 2, 3) none of which are according to the generated results. The generated results have source, target, and generations while the function expect other fields (such as tails, head, etc.)

I got this error $bash ./run.sh : AttributeError: can't set attribute

% ipython
Python 3.8.13 (default, Mar 28 2022, 06:16:26) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.2.0 -- An enhanced Interactive Python. Type '?' for help.
In [2]: import pytorch_lightning
In [3]: print(pytorch_lightning.__version__)
1.6.3

In [4]: import torch
In [5]: print(torch.__version__)
1.10.1

% bash ./run.sh           
Traceback (most recent call last):
  File "finetune.py", line 441, in <module>
    main(args)
  File "finetune.py", line 330, in main
    model: SummarizationModule = SummarizationModule(args)
  File "finetune.py", line 68, in __init__
    super().__init__(hparams, num_labels=None, mode=self.mode, **kwargs)
  File "/Users/davidlaxer/comet-atomic-2020/models/comet_atomic2020_bart/lightning_base.py", line 59, in __init__
    self.hparams = hparams
  File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1225, in __setattr__
    object.__setattr__(self, name, value)
AttributeError: can't set attribute

https://github.com/PyTorchLightning/pytorch-lightning/issues/7443

Could you please share or upload the COMET 2020 results files so that I can check the evaluation method you developed? The file automatic_eval.py in system_eval contains the mentioned experiment files

The target or tail of many relations is short, for example it could be adjectives like happy, satisfied for xReact relation. How do you measure BLEU for these tails? As I read in the following implementation of BLEU

https://github.com/mjpost/sacrebleu/issues/150#issuecomment-808509500

The sentences must have at least 4 words so that it calculate BLEU. Moreover, because of the problems mentioned in previous issues I couldn't use eval.py to evaluate the generated lists by your code

Do you have an implementation of COMET with T5? I have seen the following article mentioned it, but I couldn't find any code.

Understanding Few-Shot Commonsense Knowledge Models

Could you guide me on how I can change the current code to use T5? I've seen you declared T5 in some codes but you didn't use that.

Thanks

Hi, I'm quite interested in the result of applying COMET on my own dataset. I've also checked your demo and found that I can do Commonsense Inferences about People and Events after entering a sentence and click 'predict'. But now I want to do inferences for plenty sentences, automatically, and in this case what should I do to edit corresponding py files to reach my goal? Looking forward to your replies!

When I tried to run gpt2_zeroshot.py, the following error occurred:

FileNotFoundError: [Errno 2] No such file or directory: 'data/gpt2-zeroshot/atomic/test_samp
led_prefixes.jsonl'

Where is this file, or how I can create it?

Hi! When I installed from the requirements.txt and ran the test command:

python -m models.comet_atomic2020_gpt2.comet_gpt2 --test_install

It showed that it required transformers and wandb.

I'd like to use 'comet' with the new PyTorch 'mps' backend. The PyTorch 'mps' implementation is still being 'shaken-out', as is PyTorch-Lightning 1.6.4, however, some of the code works. Can you create a development branch of 'comet' that can use the latest version of 'PyTorch-Lightning' so we can run on the 'mps' backend (GPU).

Current comet model often requires X wanted to as a prefix. Without such prefix, the model output are not good enough in some cases.

We might want to emphasize the prefix, or to retrain the model which works well without such prefix.

consider sent1="X drives too fast" and sent2="X is pulled over by a cop" Now we know "sent1 "happens before" sent2" is true. Is there any zero-shot way of finding out whether this is true or not? Also What if in ATOMIC, we have sent1 -> r1 -> r2 -> ... -> rk -> sent2? Is there a way to find out about this from COMET? I don't want to know this from ATOMIC because well sent1 and sent2 can be sentences outside of ATOMIC. That's where COMET would be useful.