PyTorch code for JEREX: Joint Entity-Level Relation Extractor

Related tags

Deep Learning relation-extraction document-level joint-learning multi-instance-learning
Overview

JEREX: "Joint Entity-Level Relation Extractor"

PyTorch code for JEREX: "Joint Entity-Level Relation Extractor". For a description of the model and experiments, see our paper "An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning": https://arxiv.org/abs/2102.05980 (accepted at EACL 2021).

alt text

Setup

Requirements

  • Required
    • Python 3.7+
    • PyTorch (tested with version 1.8.1 - see here on how to install the correct version)
    • PyTorch Lightning (tested with version 1.2.7)
    • transformers (+sentencepiece, e.g. with 'pip install transformers[sentencepiece]', tested with version 4.5.1)
    • hydra-core (tested with version 1.0.6)
    • scikit-learn (tested with version 0.21.3)
    • tqdm (tested with version 4.43.0)
    • numpy (tested with version 1.18.1)
    • jinja2 (tested with version 2.11.3)

Fetch data

Execute the following steps before running the examples.

(1) Fetch end-to-end (joint) DocRED [1] dataset split. For the original DocRED split, see https://github.com/thunlp/DocRED :

bash ./scripts/fetch_datasets.sh

(2) Fetch model checkpoints (joint multi-instance model (end-to-end split) and relation classification multi-instance model (original split)):

bash ./scripts/fetch_models.sh

Examples

End-to-end (joint) model

(1) Train JEREX (joint model) using the end-to-end split:

python ./jerex_train.py --config-path configs/docred_joint

(2) Evaluate JEREX (joint model) on the end-to-end split (you need to fetch the model first):

python ./jerex_test.py --config-path configs/docred_joint

Relation Extraction (only) model

To run these examples, first download the original DocRED dataset into './data/datasets/docred/' (see 'https://github.com/thunlp/DocRED' for instructions)

(1) Train JEREX (multi-instance relation classification component) using the orignal DocRED dataset.

python ./jerex_train.py --config-path configs/docred

(2) Evaluate JEREX (multi-instance relation classification component) on the original DocRED test set (you need to fetch the model first):

python ./jerex_test.py --config-path configs/docred

Since the original test set labels are hidden, the code will output an F1 score of 0. A 'predictions.json' file is saved, which can be used to retrieve test set metrics by uploading it to the DocRED CodaLab challenge (see https://github.com/thunlp/DocRED)

Reproduction and Evaluation

  • If you want to compare your end-to-end model to JEREX using the strict evaluation setting, have a look at our evaluation script.
  • The DocRED dataset contains some duplicate annotations (especially entity mentions). Duplicates are removed during evaluation (i.e. only counted once).

Configuration / Hyperparameters

  • The hyperparameters used in our paper are set as default. You can adjust hyperparameters and other configuration settings in the 'train.yaml' and 'test.yaml' under ./configs
  • Settings can also be overriden via command line, e.g.:
python ./jerex_train.py training.max_epochs=40
  • A brief explanation of available configuration settings can be found in './configs.py'
  • Besides the main JEREX model ('joint_multi_instance') and the 'global' baseline ('joint_global') you can also train each sub-component ('mention_localization', 'coreference_resolution', 'entity_classification', 'relation_classification_multi_instance', 'relation_classification_global') individually. Just set 'model.model_type' accordingly (e.g. 'model.model_type: joint_global')

Prediction result inspection / Postprocessing

  • When testing a model ('./jerex_test.py') or by either specifying a test dataset (using 'datasets.test_path' configuration) or setting 'final_valid_evaluate' to True (using 'misc.final_valid_evaluate=true' configuration) during training ('./jerex_train.py'), a file containing the model's predictions is stored ('predictions.json').
  • By using a joint model ('joint_multi_instance' / 'joint_global'), a file ('examples.html') containing visualizations of all prediction results is also stored alongside 'predictions.json'.

Training/Inference speed and memory consumption

Performing a search over token spans (and pairs of spans) in the input document (as in JEREX) can be quite (CPU/GPU) memory demanding. If you run into memory issues (i.e. crashing of training/inference), these settings may help:

  • 'training.max_spans'/'training.max_coref_pairs'/'training.max_rel_pairs' (or 'inference.max_spans'/'inference.max_coref_pairs'/'inference.max_rel_pairs'): These settings restrict the number of spans/mention pairs for coreference resolution/mention pairs for MI relation classification that are processed simultaneously. Setting these to a lower number reduces training/inference speed, but lowers memory consumption.
  • The default setting of maximum span size is quite large. If the entity mentions in your dataset are usually shorter than 10 tokens, you can restrict the span search to less tokens (by setting 'sampling.max_span_size')

References

[1] Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin,Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou,and Maosong Sun. 2019.  DocRED: A Large-Scale Document-Level  Relation  Extraction  Dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764–777, Florence, Italy. ACL.
Comments
  • train problem

    train problem

    I don't know why? I didn't edit any code when running 'python ./jerex_train.py --config-path configs/docred_joint'. C:\Users\Administrator.conda\envs\torchGPU\python.exe F:/pythonproject/jerex-main/jerex_train.py --config-path configs/docred_joint F:/pythonproject/jerex-main/jerex_train.py:24: UserWarning: 'train' is validated against ConfigStore schema with the same name. This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2. See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions. train() datasets: train_path: ./data/datasets/docred_joint/train_joint.json valid_path: ./data/datasets/docred_joint/dev_joint.json test_path: null types_path: ./data/datasets/docred_joint/types.json model: model_type: joint_multi_instance encoder_path: bert-base-cased tokenizer_path: bert-base-cased mention_threshold: 0.85 coref_threshold: 0.85 rel_threshold: 0.6 prop_drop: 0.1 meta_embedding_size: 25 size_embeddings_count: 30 ed_embeddings_count: 300 token_dist_embeddings_count: 700 sentence_dist_embeddings_count: 50 position_embeddings_count: 700 sampling: neg_mention_count: 200 neg_coref_count: 200 neg_relation_count: 200 max_span_size: 10 sampling_processes: 8 neg_mention_overlap_ratio: 0.5 lowercase: false loss: mention_weight: 1.0 coref_weight: 1.0 entity_weight: 0.25 relation_weight: 1.0 inference: valid_batch_size: 1 test_batch_size: 1 max_spans: null max_coref_pairs: null max_rel_pairs: null training: batch_size: 1 min_epochs: 20 max_epochs: 20 lr: 5.0e-05 lr_warmup: 0.1 weight_decay: 0.01 max_grad_norm: 1.0 accumulate_grad_batches: 1 max_spans: null max_coref_pairs: null max_rel_pairs: null distribution: gpus:

    • 0 accelerator: '' prepare_data_per_node: false misc: store_predictions: true store_examples: true flush_logs_every_n_steps: 1000 log_every_n_steps: 1000 deterministic: false seed: null cache_path: null precision: 32 profiler: null final_valid_evaluate: true

    [2021-12-29 20:27:07,884][numexpr.utils][INFO] - NumExpr defaulting to 6 threads. Parse dataset 'F:\pythonproject\jerex-main\data\datasets\docred_joint\train_joint.json': 100%|██████████| 3008/3008 [00:48<00:00, 61.77it/s] Parse dataset 'F:\pythonproject\jerex-main\data\datasets\docred_joint\dev_joint.json': 100%|██████████| 300/300 [00:04<00:00, 67.58it/s] Some weights of the model checkpoint at bert-base-cased were not used when initializing JointMultiInstanceModel: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight']

    • This IS expected if you are initializing JointMultiInstanceModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

    • This IS NOT expected if you are initializing JointMultiInstanceModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of JointMultiInstanceModel were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['coreference_resolution.coref_classifier.bias', 'entity_classification.entity_classifier.weight', 'coreference_resolution.coref_linear.bias', 'entity_classification.linear.bias', 'relation_classification.token_distance_embeddings.weight', 'mention_localization.linear.weight', 'entity_classification.linear.weight', 'mention_localization.linear.bias', 'relation_classification.entity_type_embeddings.weight', 'relation_classification.rel_classifier.bias', 'entity_classification.entity_classifier.bias', 'coreference_resolution.coref_linear.weight', 'relation_classification.rel_linear.weight', 'relation_classification.sentence_distance_embeddings.weight', 'mention_localization.mention_classifier.weight', 'relation_classification.rel_classifier.weight', 'coreference_resolution.coref_classifier.weight', 'relation_classification.pair_linear.weight', 'mention_localization.mention_classifier.bias', 'mention_localization.size_embeddings.weight', 'coreference_resolution.coref_ed_embeddings.weight', 'relation_classification.pair_linear.bias', 'relation_classification.rel_linear.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. GPU available: True, used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

      | Name | Type | Params

    0 | model | JointMultiInstanceModel | 113 M

    113 M Trainable params 0 Non-trainable params 113 M Total params 455.954 Total estimated model params size (MB) Epoch 0: 0%| | 3/3308 [00:20<6:22:32, 6.94s/it, loss=69.4, v_num=0_0]Error executing job with overrides: [] Traceback (most recent call last): File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 637, in run_train self.train_loop.run_training_epoch() File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 492, in run_training_epoch batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 654, in run_training_batch self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 433, in optimizer_step using_lbfgs=is_lbfgs, File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\core\lightning.py", line 1390, in optimizer_step optimizer.step(closure=optimizer_closure) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\core\optimizer.py", line 214, in step self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\core\optimizer.py", line 134, in __optimizer_step trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 277, in optimizer_step self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 282, in run_optimizer_step self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 163, in optimizer_step optimizer.step(closure=lambda_closure, **kwargs) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\torch\optim\lr_scheduler.py", line 67, in wrapper return wrapped(*args, **kwargs) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\transformers\optimization.py", line 321, in step loss = closure() File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 649, in train_step_and_backward_closure split_batch, batch_idx, opt_idx, optimizer, self.trainer.hiddens File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 742, in training_step_and_backward result = self.training_step(split_batch, batch_idx, opt_idx, hiddens) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 293, in training_step training_step_output = self.trainer.accelerator.training_step(args) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 156, in training_step return self.training_type_plugin.training_step(*args) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 125, in training_step return self.lightning_module.training_step(*args, **kwargs) File "F:\pythonproject\jerex-main\jerex\model.py", line 114, in training_step outputs = self(**batch) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "F:\pythonproject\jerex-main\jerex\model.py", line 106, in forward max_rel_pairs=max_rel_pairs, inference=inference) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "F:\pythonproject\jerex-main\jerex\models\joint_models.py", line 142, in forward return self._forward_train(*args, **kwargs) File "F:\pythonproject\jerex-main\jerex\models\joint_models.py", line 198, in _forward_train rel_entity_types, max_pairs=max_rel_pairs) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "F:\pythonproject\jerex-main\jerex\models\modules\relation_classification_multi_instance.py", line 49, in forward chunk_rel_sentence_distances, mention_reprs, chunk_h) File "F:\pythonproject\jerex-main\jerex\models\modules\relation_classification_multi_instance.py", line 73, in _create_mention_pair_representations rel_ctx = m + h RuntimeError: CUDA out of memory. Tried to allocate 450.00 MiB (GPU 0; 4.00 GiB total capacity; 2.46 GiB already allocated; 168.61 MiB free; 2.79 GiB reserved in total by PyTorch)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "F:/pythonproject/jerex-main/jerex_train.py", line 20, in train model.train(cfg) File "F:\pythonproject\jerex-main\jerex\model.py", line 341, in train trainer.fit(model, datamodule=data_module) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 499, in fit self.dispatch() File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 546, in dispatch self.accelerator.start_training(self) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 73, in start_training self.training_type_plugin.start_training(trainer) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 114, in start_training self._results = trainer.run_train() File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 670, in run_train self.train_loop.on_train_end() File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 134, in on_train_end self.check_checkpoint_callback(should_update=True, is_last=True) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 164, in check_checkpoint_callback cb.on_validation_end(self.trainer, model) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 212, in on_validation_end self.save_checkpoint(trainer, pl_module) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 247, in save_checkpoint self._validate_monitor_key(trainer) File "C:\Users\Administrator.conda\envs\torchGPU\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 495, in _validate_monitor_key raise MisconfigurationException(m) pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='valid_f1') not found in the returned metrics: ['train_mention_loss', 'train_coref_loss', 'train_entity_loss', 'train_rel_loss', 'train_loss']. HINT: Did you call self.log('valid_f1', value) in the LightningModule?

    Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. Epoch 0: 0%| | 3/3308 [00:21<6:39:12, 7.25s/it, loss=69.4, v_num=0_0]

    opened by niningliuhen2013 7
  • What's the best way to use the joint model to infer entities, and relations among them, in a fresh new phrase?

    What's the best way to use the joint model to infer entities, and relations among them, in a fresh new phrase?

    What's the best way to use the joint model to infer entities, and relations among them, in a fresh new phrase, for example for this phrase:

     " I-B is one of the sub-committees that advises the Scientific Advisory Group for Emergencies (Sage), led by Sir Patrick 
Vallance, the chief scientific adviser"

    ?

    opened by raphael10-collab 7
  • Train Help

    Train Help

    I have tried to create data by devide text into sentance from spacy and mannually anotate. After that i am trying to train the model and i am getting following error. Can you please help me here? As per me the main error is my sentence length. So is their any limit of that?

    Epoch 0: 0% 0/2 [00:00<?, ?it/s] Traceback (most recent call last): File "./jerex_train.py", line 20, in train model.train(cfg) File "/content/jerex/jerex/model.py", line 341, in train trainer.fit(model, datamodule=data_module) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit self.dispatch() File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch self.accelerator.start_training(self) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training self.training_type_plugin.start_training(trainer) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training self._results = trainer.run_train() File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 637, in run_train self.train_loop.run_training_epoch() File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 492, in run_training_epoch batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 654, in run_training_batch self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 433, in optimizer_step using_lbfgs=is_lbfgs, File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/lightning.py", line 1390, in optimizer_step optimizer.step(closure=optimizer_closure) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/optimizer.py", line 214, in step self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 277, in optimizer_step self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 282, in run_optimizer_step self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 163, in optimizer_step optimizer.step(closure=lambda_closure, **kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py", line 65, in wrapper return wrapped(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py", line 88, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py", line 318, in step loss = closure() File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 649, in train_step_and_backward_closure split_batch, batch_idx, opt_idx, optimizer, self.trainer.hiddens File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 742, in training_step_and_backward result = self.training_step(split_batch, batch_idx, opt_idx, hiddens) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 293, in training_step training_step_output = self.trainer.accelerator.training_step(args) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 156, in training_step return self.training_type_plugin.training_step(*args) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 125, in training_step return self.lightning_module.training_step(*args, **kwargs) File "/content/jerex/jerex/model.py", line 114, in training_step outputs = self(**batch) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/content/jerex/jerex/model.py", line 106, in forward max_rel_pairs=max_rel_pairs, inference=inference) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/content/jerex/jerex/models/joint_models.py", line 142, in forward return self._forward_train(*args, **kwargs) File "/content/jerex/jerex/models/joint_models.py", line 188, in _forward_train max_spans=max_spans) File "/content/jerex/jerex/models/joint_models.py", line 57, in _forward_train_common h = self.bert(input_ids=encodings, attention_mask=context_masks)['last_hidden_state'] File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 969, in forward past_key_values_length=past_key_values_length, File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 207, in forward embeddings += position_embeddings RuntimeError: The size of tensor a (1514) must match the size of tensor b (700) at non-singleton dimension 1

    Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

    opened by kbrajwani 4
  • Errors when performing tests

    Errors when performing tests

    Testing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 700/700 [05:24<00:00, 2.27it/s]Evaluation

    --- Entity Mentions ---

    Traceback (most recent call last): File "./jerex_test.py", line 20, in test model.test(cfg) File "/home/jhj/jerex/jerex/model.py", line 389, in test trainer.test(model, datamodule=data_module) File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 910, in test results = self.__test_given_model(model, test_dataloaders) File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 970, in __test_given_model results = self.fit(model) File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit self.dispatch() File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 540, in dispatch self.accelerator.start_testing(self) File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 76, in start_testing self.training_type_plugin.start_testing(trainer) File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 118, in start_testing self._results = trainer.run_test() File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 786, in run_test eval_loop_results, _ = self.run_evaluation() File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 741, in run_evaluation deprecated_eval_results = self.evaluation_loop.evaluation_epoch_end() File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 189, in evaluation_epoch_end deprecated_results = self.__run_eval_epoch_end(self.num_dataloaders) File "/home/jhj/anaconda3/envs/New_Env/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 221, in __run_eval_epoch_end eval_results = model.test_epoch_end(eval_results) File "/home/jhj/jerex/jerex/model.py", line 155, in test_epoch_end metrics = self._evaluator.compute_metrics(self._eval_test_gt, predictions) File "/home/jhj/jerex/jerex/evaluation/joint_evaluator.py", line 76, in compute_metrics mention_eval = scoring.score(gt_mentions, pred_mentions, print_results=True) File "/home/jhj/jerex/jerex/evaluation/scoring.py", line 55, in score metrics = _compute_metrics(gt_flat, pred_flat, labels, labels_str, print_results) File "/home/jhj/jerex/jerex/evaluation/scoring.py", line 64, in _compute_metrics per_type = prfs(gt_all, pred_all, labels=labels, average=None, zero_division=0) TypeError: precision_recall_fscore_support() got an unexpected keyword argument 'zero_division'

    Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. Testing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 700/700 [05:24<00:00, 2.16it/s]

    Hi. When testing, an error like that occurs and the result value is not saved. Any ideas?

    thanks

    opened by zozni 3
  • performance metrics is zero

    performance metrics is zero

    Can you please help me with the following things. I have create a custom dataset with same structure as docred and tried to train the model. First of all it's not seems training by looking at performace metrics. Also after some epochs it's gives error and stop training. you can check the following logs.

    datasets:
  train_path: ./data/datasets/docred_joint/train_joint.json
  valid_path: ./data/datasets/docred_joint/dev_joint.json
  test_path: null
  types_path: ./data/datasets/docred_joint/types.json
model:
  model_type: joint_multi_instance
  encoder_path: bert-base-cased
  tokenizer_path: bert-base-cased
  mention_threshold: 0.85
  coref_threshold: 0.85
  rel_threshold: 0.6
  prop_drop: 0.1
  meta_embedding_size: 25
  size_embeddings_count: 40
  ed_embeddings_count: 300
  token_dist_embeddings_count: 700
  sentence_dist_embeddings_count: 50
  position_embeddings_count: 700
sampling:
  neg_mention_count: 200
  neg_coref_count: 200
  neg_relation_count: 200
  max_span_size: 10
  sampling_processes: 8
  neg_mention_overlap_ratio: 0.5
  lowercase: false
loss:
  mention_weight: 1.0
  coref_weight: 1.0
  entity_weight: 0.25
  relation_weight: 1.0
inference:
  valid_batch_size: 1
  test_batch_size: 1
  max_spans: null
  max_coref_pairs: null
  max_rel_pairs: null
training:
  batch_size: 1
  min_epochs: 20
  max_epochs: 20
  lr: 5.0e-05
  lr_warmup: 0.1
  weight_decay: 0.01
  max_grad_norm: 1.0
  accumulate_grad_batches: 1
  max_spans: null
  max_coref_pairs: null
  max_rel_pairs: null
distribution:
  gpus: []
  accelerator: ''
  prepare_data_per_node: false
misc:
  store_predictions: true
  store_examples: true
  flush_logs_every_n_steps: 1000
  log_every_n_steps: 1000
  deterministic: false
  seed: null
  cache_path: null
  precision: 32
  profiler: null
  final_valid_evaluate: true

Parse dataset '/content/jerex/data/datasets/docred_joint/train_joint.json': 100% 260/260 [00:02<00:00, 111.80it/s]
Parse dataset '/content/jerex/data/datasets/docred_joint/dev_joint.json': 100% 30/30 [00:00<00:00, 120.30it/s]
Some weights of the model checkpoint at bert-base-cased were not used when initializing JointMultiInstanceModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'bert.pooler.dense.weight', 'bert.pooler.dense.bias']
- This IS expected if you are initializing JointMultiInstanceModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing JointMultiInstanceModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of JointMultiInstanceModel were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['mention_localization.linear.weight', 'mention_localization.linear.bias', 'mention_localization.mention_classifier.weight', 'mention_localization.mention_classifier.bias', 'mention_localization.size_embeddings.weight', 'coreference_resolution.coref_linear.weight', 'coreference_resolution.coref_linear.bias', 'coreference_resolution.coref_classifier.weight', 'coreference_resolution.coref_classifier.bias', 'coreference_resolution.coref_ed_embeddings.weight', 'entity_classification.linear.weight', 'entity_classification.linear.bias', 'entity_classification.entity_classifier.weight', 'entity_classification.entity_classifier.bias', 'relation_classification.pair_linear.weight', 'relation_classification.pair_linear.bias', 'relation_classification.rel_linear.weight', 'relation_classification.rel_linear.bias', 'relation_classification.rel_classifier.weight', 'relation_classification.rel_classifier.bias', 'relation_classification.token_distance_embeddings.weight', 'relation_classification.sentence_distance_embeddings.weight', 'relation_classification.entity_type_embeddings.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
GPU available: True, used: False
TPU available: False, using: 0 TPU cores
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py:52: UserWarning: GPU available but not used. Set the --gpus flag when calling the script.
  warnings.warn(*args, **kwargs)

  | Name  | Type                    | Params
--------------------------------------------------
0 | model | JointMultiInstanceModel | 113 M 
--------------------------------------------------
113 M     Trainable params
0         Non-trainable params
113 M     Total params
455.678   Total estimated model params size (MB)
Epoch 0:  90% 260/290 [05:11<00:35,  1.20s/it, loss=0.568, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)

Epoch 0: 100% 290/290 [05:20<00:00,  1.10s/it, loss=0.568, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.51it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 0: 100% 290/290 [05:24<00:00,  1.12s/it, loss=0.568, v_num=0_0]
                                               tcmalloc: large alloc 1297850368 bytes == 0x56535c070000 @  0x7fa2102d4615 0x56526cb7b4cc 0x56526cc5b47a 0x56526cb81f0c 0x7fa20ae659e4 0x7fa20ae6e0e4 0x7fa20ae42bf0 0x7fa168968935 0x7fa16896426e 0x7fa16896940a 0x7fa20ae4313e 0x7fa20a8d8f98 0x56526cb7f098 0x56526cbf24d9 0x56526cbecced 0x56526cb7fbda 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbedc0d 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbedc0d 0x56526cb7fafa
tcmalloc: large alloc 1622319104 bytes == 0x5653a962a000 @  0x7fa2102d4615 0x56526cb7b4cc 0x56526cc5b47a 0x56526cb81f0c 0x7fa20ae659e4 0x7fa20ae6e0e4 0x7fa20ae42bf0 0x7fa168968935 0x7fa16896426e 0x7fa16896940a 0x7fa20ae4313e 0x7fa20a8d8f98 0x56526cb7f098 0x56526cbf24d9 0x56526cbecced 0x56526cb7fbda 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbed915 0x56526cbec9ee 0x56526cb7fbda 0x56526cbedc0d 0x56526cbec9ee 0x56526cb7fbda 0x56526cbf1d00 0x56526cb7fafa 0x56526cbedc0d 0x56526cb7fafa
Epoch 1:  90% 260/290 [05:23<00:37,  1.24s/it, loss=0.817, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 1: 100% 290/290 [05:31<00:00,  1.14s/it, loss=0.817, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.49it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 1: 100% 290/290 [05:35<00:00,  1.16s/it, loss=0.817, v_num=0_0]
Epoch 2:  90% 260/290 [05:40<00:39,  1.31s/it, loss=0.343, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 2: 100% 290/290 [05:48<00:00,  1.20s/it, loss=0.343, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.51it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 2: 100% 290/290 [05:52<00:00,  1.22s/it, loss=0.343, v_num=0_0]
Epoch 3:  90% 260/290 [05:49<00:40,  1.34s/it, loss=0.175, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 3: 100% 290/290 [05:57<00:00,  1.23s/it, loss=0.175, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.50it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 3: 100% 290/290 [06:01<00:00,  1.25s/it, loss=0.175, v_num=0_0]
Epoch 4:  90% 260/290 [06:16<00:43,  1.45s/it, loss=0.12, v_num=0_0] 
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 4: 100% 290/290 [06:25<00:00,  1.33s/it, loss=0.12, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.39it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 4: 100% 290/290 [06:29<00:00,  1.34s/it, loss=0.12, v_num=0_0]
Epoch 5:  90% 260/290 [06:05<00:42,  1.41s/it, loss=0.117, v_num=0_0] 
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 5: 100% 290/290 [06:14<00:00,  1.29s/it, loss=0.117, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.52it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 5: 100% 290/290 [06:17<00:00,  1.30s/it, loss=0.117, v_num=0_0]
Epoch 6:  90% 260/290 [06:07<00:42,  1.41s/it, loss=0.1, v_num=0_0]  
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 6: 100% 290/290 [06:16<00:00,  1.30s/it, loss=0.1, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.47it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 6: 100% 290/290 [06:20<00:00,  1.31s/it, loss=0.1, v_num=0_0]
Epoch 7:  90% 260/290 [06:04<00:42,  1.40s/it, loss=0.153, v_num=0_0] 
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 7: 100% 290/290 [06:12<00:00,  1.29s/it, loss=0.153, v_num=0_0]
Validating: 100% 30/30 [00:12<00:00,  2.49it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 7: 100% 290/290 [06:16<00:00,  1.30s/it, loss=0.153, v_num=0_0]
Epoch 8:  90% 260/290 [06:11<00:42,  1.43s/it, loss=0.0601, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 8: 100% 290/290 [06:20<00:00,  1.31s/it, loss=0.0601, v_num=0_0]
Validating: 100% 30/30 [00:13<00:00,  2.27it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 8: 100% 290/290 [06:25<00:00,  1.33s/it, loss=0.0601, v_num=0_0]
Epoch 9:  90% 260/290 [06:21<00:44,  1.47s/it, loss=0.0318, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 9: 100% 290/290 [06:30<00:00,  1.35s/it, loss=0.0318, v_num=0_0]
Validating: 100% 30/30 [00:13<00:00,  2.25it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 9: 100% 290/290 [06:35<00:00,  1.36s/it, loss=0.0318, v_num=0_0]
Epoch 10:  90% 260/290 [06:23<00:44,  1.47s/it, loss=0.0143, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Validating:   0% 0/30 [00:00<?, ?it/s]
Epoch 10: 100% 290/290 [06:33<00:00,  1.36s/it, loss=0.0143, v_num=0_0]
Validating: 100% 30/30 [00:14<00:00,  2.15it/s]Evaluation

--- Entity Mentions ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Clusters (Coreference Resolution) ---

                type    precision       recall     f1-score      support
              Binary         0.00         0.00         0.00        111.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Entities ---

                type    precision       recall     f1-score      support
               PRICE         0.00         0.00         0.00          3.0
               ALIAS         0.00         0.00         0.00         40.0
           LIABILITY         0.00         0.00         0.00          2.0
           REFERENCE         0.00         0.00         0.00         34.0
             ADDRESS         0.00         0.00         0.00         32.0

               micro         0.00         0.00         0.00        111.0
               macro         0.00         0.00         0.00        111.0

--- Relations ---
Without entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0

With entity type

                type    precision       recall     f1-score      support
       ALIAS_ADDRESS         0.00         0.00         0.00          7.0
     LIABILITY_PRICE         0.00         0.00         0.00          3.0
     ALIAS_REFERENCE         0.00         0.00         0.00         12.0
   REFERENCE_ADDRESS         0.00         0.00         0.00         10.0

               micro         0.00         0.00         0.00         32.0
               macro         0.00         0.00         0.00         32.0
Epoch 10: 100% 290/290 [06:37<00:00,  1.37s/it, loss=0.0143, v_num=0_0]
Epoch 11:  90% 260/290 [06:21<00:44,  1.47s/it, loss=0.0324, v_num=0_0]
Validating: 0it [00:00, ?it/s]
Epoch 11:  97% 280/290 [06:24<00:13,  1.37s/it, loss=0.0324, v_num=0_0]
Traceback (most recent call last):
  File "./jerex_train.py", line 20, in train
    model.train(cfg)
  File "/content/jerex/jerex/model.py", line 341, in train
    trainer.fit(model, datamodule=data_module)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch
    self.accelerator.start_training(self)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
    self._results = trainer.run_train()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 637, in run_train
    self.train_loop.run_training_epoch()
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/training_loop.py", line 577, in run_training_epoch
    self.trainer.run_evaluation(on_epoch=True)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 725, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 166, in evaluation_step
    output = self.trainer.accelerator.validation_step(args)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 177, in validation_step
    return self.training_type_plugin.validation_step(*args)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 131, in validation_step
    return self.lightning_module.validation_step(*args, **kwargs)
  File "/content/jerex/jerex/model.py", line 126, in validation_step
    return self._inference(batch, batch_idx)
  File "/content/jerex/jerex/model.py", line 176, in _inference
    output = self(**batch, inference=True)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/jerex/jerex/model.py", line 106, in forward
    max_rel_pairs=max_rel_pairs, inference=inference)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/jerex/jerex/models/joint_models.py", line 144, in forward
    return self._forward_inference(*args, **kwargs)
  File "/content/jerex/jerex/models/joint_models.py", line 209, in _forward_inference
    mention_sample_masks, max_spans=max_spans, max_coref_pairs=max_coref_pairs)
  File "/content/jerex/jerex/models/joint_models.py", line 100, in _forward_inference_common
    valid_mentions, self._coref_threshold)
  File "/content/jerex/jerex/models/misc.py", line 135, in create_clusters
    assignments = agg_clustering.fit_predict(distances)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/cluster/_agglomerative.py", line 902, in fit_predict
    return super().fit_predict(X, y)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/base.py", line 462, in fit_predict
    self.fit(X)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/cluster/_agglomerative.py", line 796, in fit
    X = check_array(X, ensure_min_samples=2, estimator=self)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py", line 586, in check_array
    context))
ValueError: Found array with 1 sample(s) (shape=(1, 1)) while a minimum of 2 is required by AgglomerativeClustering.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
    opened by injala-kumar 3
  • Help in interpreting the test results

    Help in interpreting the test results

    Yesterday, after training the model ( https://github.com/lavis-nlp/jerex/issues/3#issuecomment-852361468 ) I tested it. Now I ask you to give me an help in interpreting the results

    I guess that the predictions made during the test by the model, which was previously trained, have been stored yesterday in predictions.json file. Is this right?

    (base) raphy@pc:~/jerex/data/runs/2021-06-01/21-32-32$ ls -lah
total 122M
drwxrwxr-x 5 raphy raphy 4,0K giu  2 08:59 .
drwxrwxr-x 3 raphy raphy 4,0K giu  1 21:32 ..
drwxrwxr-x 3 raphy raphy 4,0K giu  1 21:32 cv
-rw-rw-r-- 1 raphy raphy 120M giu  1 21:42 examples_test.html
-rw-rw-r-- 1 raphy raphy    0 giu  1 21:32 jerex_test.log
-rw-rw-r-- 1 raphy raphy 2,2M giu  1 21:42 predictions.json
drwxrwxr-x 2 raphy raphy 4,0K giu  1 21:32 run_config
drwxrwxr-x 3 raphy raphy 4,0K giu  1 21:32 tb
(base) raphy@pc:~/jerex/data/runs/2021-06-01/21-32-32$

    I loaded predictions.json file in https://jsonformatter.org/json-viewer to better view and understand it.

    Example:

    relation number 0: 
    head: 0   ----------->   entities: 0 
                                          cluster: 0                -----------------> 0:13
                                                                                                1: 17

                                                                                                Does it mean from token 13 to token 17 ?
                                                                                                 tokens: 
                                                                                                     13: -
                                                                                                     14: platform
                                                                                                     15: producer
                                                                                                     16: and
                                                                                                     17: director

                                          type: MISC

    tail: 15     -------------> entities: 15
                                           cluster: 15              -----------------> 0:12

                                                                                                 Does it mean token 12 ?
                                                                                                    tokens:
                                                                                                        12: cross
                                           type: TIME
    type: P577

    Does it mean that it found a relation of type TIME between "platform producer and director" and "cross"?

    opened by raphael10-collab 2
  • Question regarding

    Question regarding "long" documents

    Hi @markus-eberts ! Congratulations for your work!

    I have a question... I was trying to debug your code, and didn't quite find the point in which text sequences longer than 512 tokens are being fed to bert / transformer model. How exactly are you handling this? Where in the code is this?

    Thanks!

    opened by pvcastro 2
  • RuntimeError: [enforce fail at CPUAllocator.cpp:67] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 42472464384 bytes. Error code 12 (Cannot allocate memory)

    RuntimeError: [enforce fail at CPUAllocator.cpp:67] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 42472464384 bytes. Error code 12 (Cannot allocate memory)

    During the training process I'm getting this error:

    RuntimeError: [enforce fail at CPUAllocator.cpp:67] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 42472464384 bytes. Error code 12 (Cannot allocate memory)

    Epoch 5:  98%|███████████████████████████████████████████████████████████████████████████████████████████████▎ | 3250/3308 [1:10:30<01:15,  1.30s/it, loss=0.165, v_num=0_0]Traceback (most recent call last):████████████████████████████████████████████████████████████████████████████████▏                       | 241/300 [02:57<00:39,  1.50it/s]
  File "./jerex_train.py", line 20, in train
    model.train(cfg)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/model.py", line 341, in train
    trainer.fit(model, datamodule=data_module)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
    self._run(model)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
    self.dispatch()
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
    self.accelerator.start_training(self)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
    return self.run_train()
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 869, in run_train
    self.train_loop.run_training_epoch()
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 576, in run_training_epoch
    self.trainer.run_evaluation(on_epoch=True)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
    output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 174, in evaluation_step
    output = self.trainer.accelerator.validation_step(args)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 226, in validation_step
    return self.training_type_plugin.validation_step(*args)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in validation_step
    return self.lightning_module.validation_step(*args, **kwargs)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/model.py", line 126, in validation_step
    return self._inference(batch, batch_idx)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/model.py", line 176, in _inference
    output = self(**batch, inference=True)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/model.py", line 106, in forward
    max_rel_pairs=max_rel_pairs, inference=inference)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/models/joint_models.py", line 144, in forward
    return self._forward_inference(*args, **kwargs)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/models/joint_models.py", line 233, in _forward_inference
    max_pairs=max_rel_pairs)
  File "/home/marco/.pyenv/versions/PyTorch1.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/models/modules/relation_classification_multi_instance.py", line 49, in forward
    chunk_rel_sentence_distances, mention_reprs, chunk_h)
  File "/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/jerex/models/modules/relation_classification_multi_instance.py", line 73, in _create_mention_pair_representations
    rel_ctx = m + h
RuntimeError: [enforce fail at CPUAllocator.cpp:67] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 42472464384 bytes. Error code 12 (Cannot allocate memory)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Epoch 5:  98%|███████████████████████████████████████████████████████████████████████████████████████████████▎ | 3250/3308 [1:10:48<01:15,  1.31s/it, loss=0.165, v_num=0_0]

    This is the memory footprint:

     (PyTorch1.7) (base) marco@pc:~/PyTorchMatters/EntitiesRelationsExtraction/jerex$ free -m                                                                                     
              total        used        free      shared  buff/cache   available
Mem:          32059        1670       20587         110        9802       29827
Swap:           979           0         979

    How much memory is required and what the minimum requirements (memory, cpu, storage,....) for running the training process? Which Google Cloud Architecture could be better suited? https://cloud.google.com/tpu/docs/tpus#when_to_use_tpus Do you think that Google's TPU are a good fit for Jerex Model's tensors shapes and dimensions? https://cloud.google.com/tpu/docs/tpus#shapes

    opened by raphael10-collab 2
  • You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference

    You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference

    When trying to train the model with python ./jerex_train.py --config-path configs/docred_joint I get this message You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. What does it mean? How should I then train the model instead?

    O.S: Ubuntu 18.04.4

    /jerex$ python ./jerex_train.py --config-path configs/docred_joint
datasets:
  train_path: ./data/datasets/docred_joint/train_joint.json
  valid_path: ./data/datasets/docred_joint/dev_joint.json
  test_path: null
  types_path: ./data/datasets/docred_joint/types.json
model:
  model_type: joint_multi_instance
  encoder_path: bert-base-cased
  tokenizer_path: bert-base-cased
  mention_threshold: 0.85
  coref_threshold: 0.85
  rel_threshold: 0.6
  prop_drop: 0.1
  meta_embedding_size: 25
  size_embeddings_count: 30
  ed_embeddings_count: 300
  token_dist_embeddings_count: 700
  sentence_dist_embeddings_count: 50
  position_embeddings_count: 700
sampling:
  neg_mention_count: 200
  neg_coref_count: 200
  neg_relation_count: 200
  max_span_size: 10
  sampling_processes: 8
  neg_mention_overlap_ratio: 0.5
  lowercase: false
loss:
  mention_weight: 1.0
  coref_weight: 1.0
  entity_weight: 0.25
  relation_weight: 1.0
inference:
  valid_batch_size: 1
  test_batch_size: 1
  max_spans: null
  max_coref_pairs: null
  max_rel_pairs: null
training:
  batch_size: 1
  min_epochs: 20
  max_epochs: 20
  lr: 5.0e-05
  lr_warmup: 0.1
  weight_decay: 0.01
  max_grad_norm: 1.0
  accumulate_grad_batches: 1
  max_spans: null
  max_coref_pairs: null
  max_rel_pairs: null
distribution:
  gpus: []
  accelerator: ''
  prepare_data_per_node: false
misc:
  store_predictions: true
  store_examples: true
  flush_logs_every_n_steps: 1000
  log_every_n_steps: 1000
  deterministic: false
  seed: null
  cache_path: null
  precision: 32
  profiler: null
  final_valid_evaluate: true

Parse dataset '/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/data/datasets/docred_joint/train_joint.json': 100%|██████| 3008/3008 [00:41<00:00, 71.72it/s]
Parse dataset '/home/marco/PyTorchMatters/EntitiesRelationsExtraction/jerex/data/datasets/docred_joint/dev_joint.json': 100%|██████████| 300/300 [00:03<00:00, 75.22it/s]
Some weights of the model checkpoint at bert-base-cased were not used when initializing JointMultiInstanceModel: ['cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'bert.pooler.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'bert.pooler.dense.bias', 'cls.predictions.transform.dense.bias']
- This IS expected if you are initializing JointMultiInstanceModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing JointMultiInstanceModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of JointMultiInstanceModel were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['entity_classification.entity_classifier.weight', 'relation_classification.pair_linear.weight', 'coreference_resolution.coref_classifier.weight', 'relation_classification.rel_classifier.weight', 'mention_localization.size_embeddings.weight', 'relation_classification.rel_linear.weight', 'mention_localization.linear.weight', 'relation_classification.sentence_distance_embeddings.weight', 'relation_classification.token_distance_embeddings.weight', 'coreference_resolution.coref_linear.bias', 'coreference_resolution.coref_linear.weight', 'mention_localization.mention_classifier.weight', 'entity_classification.linear.bias', 'mention_localization.linear.bias', 'relation_classification.entity_type_embeddings.weight', 'entity_classification.linear.weight', 'relation_classification.rel_linear.bias', 'relation_classification.rel_classifier.bias', 'coreference_resolution.coref_ed_embeddings.weight', 'coreference_resolution.coref_classifier.bias', 'entity_classification.entity_classifier.bias', 'mention_localization.mention_classifier.bias', 'relation_classification.pair_linear.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
GPU available: False, used: False
TPU available: False, using: 0 TPU cores

  | Name  | Type                    | Params
--------------------------------------------------
0 | model | JointMultiInstanceModel | 113 M 
--------------------------------------------------
113 M     Trainable params
0         Non-trainable params
113 M     Total params
455.954   Total estimated model params size (MB)
    opened by raphael10-collab 2
  • Multi-GPU training

    Multi-GPU training

    Hi all, I'm trying to run a multi-gpu training by running the following command: CUDA_VISIBLE_DEVICES=1,3 python ./jerex_train.py --config-path configs/docred_joint after the run is launched I see memory allocation for device 0 (i.e. CUDA 1), but not on device 1. I have tried with batch_size > 1 as well.

    I guess some modifications are needed in the cfg file, specifically in the following section:

    distribution:
  gpus: [0, 1]
  accelerator: ''
  prepare_data_per_node: false

    How can that be solved? My env is as described in your requirements.txt file.

    Thanks.

    opened by AvivSham 1
  • What's the relation, if there is any relation, between Jerex Model and Spert Model ?

    What's the relation, if there is any relation, between Jerex Model and Spert Model ?

    What's the relation, if there is any relation, between Jerex Model and Spert Model (https://github.com/lavis-nlp/spert) ?

    At first sight, they look similar, and may be overlapping. But I guess I'm pretty wrong

    opened by raphael10-collab 1
  • Relation Extraction metric

    Relation Extraction metric

    Hi there, Congrats on the nice work. I am one of the authors of REBEL (https://github.com/Babelscape/rebel) and we compared our work with yours on the DocRED end-to-end Relation Extraction setting.

    Due to a recent issue in our repo (https://github.com/Babelscape/rebel/issues/26), I realized that our comparison may be unfair to your work. Basically, in our work we considered the first mention to an entity in order to extract the triplets in a seq2seq fashion. I had assumed that in JEREX, a relation was correct if both entities were correct but didn't occur to me that you considered the entities correct only if all mentions are extracted.

    Ie. in your example from Table 3, would it be considered correct if the last mention to Melville wouldn't have been predicted?

    If not, then I believe it is unfair to directly compare REBEL with JEREX, as your evaluation is stricter, and I would try to clarify it.

    Thanks, Pere-Lluis.

    opened by LittlePea13 0
  • MisconfigurationException: `train_dataloader` must be implemented to be used with the Lightning Trainer

    MisconfigurationException: `train_dataloader` must be implemented to be used with the Lightning Trainer

    Hi, I tried to run jerex_train.py in a Kaggle environment using the default configurations (except distribution.gpus=[0] distribution.prepare_data_per_node=true, the latter due to a conflict error with DataModule configuration), and got the following error:

    Error executing job with overrides: ['distribution.gpus=[0]', 'distribution.prepare_data_per_node=true']
Traceback (most recent call last):
  File "jerex_train.py", line 20, in train
    model.train(cfg)
  File "/kaggle/working/jerex/model.py", line 341, in train
    trainer.fit(model, datamodule=data_module)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 769, in fit
    self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1215, in _run
    self.strategy.setup(self)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/strategies/single_device.py", line 72, in setup
    super().setup(trainer)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/strategies/strategy.py", line 139, in setup
    self.setup_optimizers(trainer)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/strategies/strategy.py", line 129, in setup_optimizers
    self.lightning_module
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/optimizer.py", line 180, in _init_optimizers_and_lr_schedulers
    optim_conf = model.trainer._call_lightning_module_hook("configure_optimizers", pl_module=model)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1593, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/kaggle/working/jerex/model.py", line 194, in configure_optimizers
    dataloader = self.train_dataloader()
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/hooks.py", line 494, in train_dataloader
    raise MisconfigurationException("`train_dataloader` must be implemented to be used with the Lightning Trainer")
pytorch_lightning.utilities.exceptions.MisconfigurationException: `train_dataloader` must be implemented to be used with the Lightning Trainer

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
    opened by famube 0
  • verbose message to increase size_embeddings_count

    verbose message to increase size_embeddings_count

    Hi First, thank you for the great effort, I learned a lot from you.
    I tried to use your model for my own dataset, but since the length of entity spans are a bit larger than the default size_embeddings_count in the config, I wasn't successful. I was getting this error message, which wasn't clear enough.

    IndexError: index out of range in self

    It took me a whole day to dig the bug up, I tried to change your code to have a more verbose message about this issue.
    I hope it can help others with a similar problem.

    opened by e3oroush 2
Owner
LAVIS - NLP Working Group
LAVIS - NLP Working Group
GitHub
Video-face-extractor - Video face extractor with Python

Python face extractor Setup Create the srcvideos and faces directories Put your

null 2 Feb 3, 2022
Metadata-Extractor - Metadata Extractor Script can be used to read in exif metadata

Metadata Extractor The exifextract script can be used to read in exif metadata f

null 1 Feb 16, 2022
Code for the paper "Relation of the Relations: A New Formalization of the Relation Extraction Problem"

This repo contains the code for the EMNLP 2020 paper "Relation of the Relations: A New Paradigm of the Relation Extraction Problem" (Jin et al., 2020)

YYY 27 Oct 26, 2022
Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs This is an implemetation of the paper Few-shot Relation Extraction via Baye

MilaGraph 36 Nov 22, 2022
Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning".

ERICA Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive L

THUNLP 75 Nov 2, 2022
Source code for "UniRE: A Unified Label Space for Entity Relation Extraction.", ACL2021.

UniRE Source code for "UniRE: A Unified Label Space for Entity Relation Extraction.", ACL2021. Requirements python: 3.7.6 pytorch: 1.8.1 transformers:

Wang Yijun 109 Nov 29, 2022
Source code for "Pack Together: Entity and Relation Extraction with Levitated Marker"

PL-Marker Source code for Pack Together: Entity and Relation Extraction with Levitated Marker. Quick links Overview Setup Install Dependencies Data Pr

THUNLP 173 Dec 30, 2022
Source code for paper "Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling", AAAI 2021

ATLOP Code for AAAI 2021 paper Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling. If you make use of this co

Wenxuan Zhou 146 Nov 29, 2022
Code for technical report "An Improved Baseline for Sentence-level Relation Extraction".

RE_improved_baseline Code for technical report "An Improved Baseline for Sentence-level Relation Extraction". Requirements torch >= 1.8.1 transformers

Wenxuan Zhou 74 Nov 29, 2022
[ACL 20] Probing Linguistic Features of Sentence-level Representations in Neural Relation Extraction

REval Table of Contents Introduction Overview Requirements Installation Probing Usage Citation License ?? Introduction REval is a simple framework for

null 13 Jan 6, 2023
It's a implement of this paper：Relation extraction via Multi-Level attention CNNs

Relation Classification via Multi-Level Attention CNNs It's a implement of this paper：Relation Classification via Multi-Level Attention CNNs. Training

Aybss 2 Nov 4, 2022
[EMNLP 2021] MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

MuVER This repo contains the code and pre-trained model for our EMNLP 2021 paper: MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity

null 24 May 30, 2022
One implementation of the paper "DMRST: A Joint Framework for Document-Level Multilingual RST Discourse Segmentation and Parsing".

Introduction One implementation of the paper "DMRST: A Joint Framework for Document-Level Multilingual RST Discourse Segmentation and Parsing". Users

seq-to-mind 18 Dec 11, 2022
Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch

Transformer in Transformer Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image c

Phil Wang 272 Dec 23, 2022
Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

DeepPanoContext (DPC) [Project Page (with interactive results)][Paper] DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context G

Cheng Zhang 66 Nov 16, 2022
Joint Versus Independent Multiview Hashing for Cross-View Retrieval[J] (IEEE TCYB 2021, PyTorch Code)

Thanks to the low storage cost and high query speed, cross-view hashing (CVH) has been successfully used for similarity search in multimedia retrieval. However, most existing CVH methods use all views to learn a common Hamming space, thus making it difficult to handle the data with increasing views or a large number of views.

null 4 Nov 19, 2022
The Pytorch code of "Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification", CVPR 2022 (Oral).

DeepBDC for few-shot learning        Introduction In this repo, we provide the implementation of the following paper: "Joint Distribution Matters: Dee

FeiLong 116 Dec 19, 2022
PyTorch implementation of ARM-Net: Adaptive Relation Modeling Network for Structured Data.

A ready-to-use framework of latest models for structured (tabular) data learning with PyTorch. Applications include recommendation, CRT prediction, healthcare analytics, and etc.

null 48 Nov 30, 2022
Pytorch Implementation for CVPR2018 Paper: Learning to Compare: Relation Network for Few-Shot Learning

LearningToCompare Pytorch Implementation for Paper: Learning to Compare: Relation Network for Few-Shot Learning Howto download mini-imagenet and make

Jackie Loong 246 Dec 19, 2022