Hi Zexuan,

The paper reports the average Rel F1 over 5 runs. Do the 5 RE runs use the same ent_pred_test.json (e.g. from the BERT (cross, W=300) entity model) or they use different NER predictions from NER models with different seeds?

Could you please share the ent_pred_test.json for the 5 RE runs?

Thanks!

Best, Deming

Hello, I have successfully converted my json output of Inception to jsonl format acceptable to your package. The output is as follows:

{"doc_key": "Test_001", "sentences": [["Google", "was", "founded", "on", "September", "4", ",", "1998", ",", "by", "Larry", "Page", "and", "Sergey", "Brin", "while", "they", "were", "Ph.D", ".", "students", "at", "Stanford", "University", "in", "California", "."], ["On", "December", "3", ",", "2019", ",", "Pichai", "also", "became", "the", "CEO", "of", "Google", "."], ["In", "March", "1999", ",", "the", "Google", "moved", "its", "offices", "to", "Palo", "Alto", ",", "California", "."]], "ner": [[[0, 0, "ORG"], [10, 11, "PER"], [13, 14, "PER"]], [[33, 33, "PER"], [39, 39, "ORG"]], [[46, 46, "ORG"], [51, 54, "LOC"]]], "relations": [[[0, 0, 10, 11, "Founder"], [0, 0, 13, 14, "Founder"]], [[33, 33, 39, 39, "CEO"]], [[46, 46, 51, 54, "Located in"]]]}

This was saved as train.json, test.json and dev.json. I wanted to test your package using this input file to see if it is working. I used the following command:

python run_entity.py --do_train --learning_rate=1e-5 --task_learning_rate=5e-4 --train_batch_size=1 --context_window 300 --data_dir data_dir --model bert-base-uncased --output_dir output_dir --task ace05

I got the following error:

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 2.00 GiB total capacity; 1.28 GiB already allocated; 5.61 MiB free; 1.39 GiB reserved in total by PyTorch)

How can I fix this? Also, do you have any recommendations for the options of my command(s) given my goal? Thank you very much.

Have you tried the argument of add_new_tokens? I used bert-base-chinese to train my chinese dataset, and the number of entity types exceeds the total [unused], after applying add_marker_tokens function, the vocab size and tokenizer length both added from 21128 to 21300. Then, I applied the origin BERT model, and applied resize_token_embeddings(), however, I found that the special token, such as <subj_start=地点>, <obj_start=人物>, etc, are all represented as 100, namely [UNK] in the sequence of input_ids after applying tokenizer.convert_tokens_to_ids(). Since the different marker tokens should have different id according to the paper, could you please help me to solve this problem?

Hello! I'm trying to use your repo but first I need to convert my annotations in json format. My annotations are in .annformat and this is a sample of them: T0 Claim 2992 3010 here are the facts T1 Premise 3012 3078 210,000 dead people in our country in just the last several months R0 Support Arg1:T0 Arg2:

Now, I am trying to create the .jsonfile to "feed" the model. My concerns are about the indexes I have to put in the .json input file for entities and relations. The indexes next to the entities "T" refer to the char indexes based on the main .txtfile.

My issue is about if I have to do some math to calculate the "new" indexes considering some offset, or the "real" indexes can be used without any problems.

Thank you in advance for your time and help. I'm trying to test your performing models with my annotations but I want to be sure the input is good.

Pierpaolo

Hi, I have used the brat tool to annotate my data, so I have two files: .ann files with the annotations of entities and relations, and raw text files. My goal is to convert these two files into the Scierc format, and finally used your pre-trained model on my datasets. However, I am having problem with the data conversion into the Scierc format.

Please can you point me to good direction on how to achieve this so I can run your model on my data?

Thanks

Hello,

For multiple entity pairs in one sentence, do you separately generate a new sentence for each entity pair during training? How do you sample negative samples? I mean entity pairs without relations

Thank you

I was wondering if you download the training data, in original sciERC format (as shown here) and then reformat again automatically internally before training the model? Asking this, because a little confused whether to format my custom dataset like sciERC or your input data format, as shown on the Readme.md . Also, sciERC annotates sentence wise , as per the aforementioned link. How does pure handle multisentence passages? The pretrained models that are downloaded by the repo-provided Readme.md link, are also labeled as single sentence model.

Hi, I was wondering what specifically is referred to by --train_file? Is it the JSONL file used as the input file for the entity model? Thank you very much.

Hello, I keep getting the error mentioned on the title while running this script :

python run_relation.py   --task scierc   --do_train --train_file scierc_data/processed_data/json/train.json   --do_eval --eval_with_gold   --model  allenai/scibert_scivocab_uncased   --do_lower_case   --train_batch_size 8   --eval_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --context_window 100   --max_seq_length 128    --entity_output_dir scierc_models/ent-scib-ctx0   --output_dir scierc_outputs

I have also tried with --eval_test argument and without using either --eval_test and --eval_with_gold, both throw the same error.
Earlier I had managed to run the first two evaluation scripts provided with the repo, seamlessly. They produced ent_pred_test.json and predictions.json, respectively and as expected. :

# Run the pre-trained entity model, the result will be stored in ${scierc_ent_model}/ent_pred_test.json
python run_entity.py \
   --do_eval --eval_test \
   --context_window 0 \
   --task scierc \
   --data_dir ${scierc_dataset} \
   --model allenai/scibert_scivocab_uncased \
   --output_dir ${scierc_ent_model}

# Run the pre-trained full relation model
python run_relation.py \
 --task scierc \
 --do_eval --eval_test \
 --model allenai/scibert_scivocab_uncased \
 --do_lower_case \
 --context_window 0\
 --max_seq_length 128 \
 --entity_output_dir ${scierc_ent_model} \
 --output_dir ${scierc_rel_model}

Could you please tell me if I am doing anything wrong?

Hello, I'm really interested in your pipelined approach in this paper. I notice that you mentioned a second pre-trained encoder in the relation model. I'm a little confused about that. I'm not sure how you get the second pre-trained encoder. Is the training procedure completely identical with BERT? Or is there anything special in getting the encoder?

Thanks,

I have successfully trained a model using custom data. Now I want to input a document and have the model predict the relations in the document. What is the command to accomplish this? Thank you very much.

Hello. Don't know if I get it right but for approximation model, why the training phase is different from the inference phase? For training phase, seems like literally it just uses one pair of entity for one instance while for inference it will list all the possible entity pairs. What's the purpose of this? And for training phase, why the label's length is padded to max_seq_length // 4? Thank you.

Implemented changes primarily in shared/const.py to enable auto-populating NER and REL labels based on data (apart from the standard labels present for ace04, ace05 and scierc). Changes include:

• Auto-populating labels when task is not one of the 3 standards (Removal of choice in task parameter)
• Storing the new-found labels in output dir at the time of entity model run for relation and relation-approx models
• Converted task labels into a class for better management
• Using the pre-populated labels in case task belongs to one of the 3 standards (for consistent replication of paper results)

Hi, I'd run into "RuntimeError: CUDA error: device-side assert triggered" error when I attempted to run your code on a Chinese dataset. The log is as follows:

/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [165,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [165,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "run_entity.py", line 225, in <module>
    output_dict = model.run_batch(train_batches[i], training=True)
  File "/tf_group/lihongyu/PURE-main/entity/models.py", line 302, in run_batch
    attention_mask = attention_mask_tensor.to(self._model_device),
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/tf_group/lihongyu/PURE-main/entity/models.py", line 65, in forward
    spans_embedding = self._get_span_embeddings(input_ids, spans, token_type_ids=token_type_ids, 
attention_mask=attention_mask)
  File "/tf_group/lihongyu/PURE-main/entity/models.py", line 41, in _get_span_embeddings
    sequence_output, pooled_output = self.bert(input_ids=input_ids, token_type_ids=token_type_ids,             
attention_mask=attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 752, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 181, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: CUDA error: device-side assert triggered

I have searched a few cases of this error on stack overflow, but still fail to make out what has happened. I drew the dimension of inputs_embeds, position_embeddings, token_type_embeddings, and it seemed to be nothing wrong(all of these is of [1, seq_len(>350), 768]) Thanks for your time.

Hi,

I'm looking to apply your pretrained models to an unlabeled, new dataset. I have my dataset in DyGIE format. Looking at the script, it's unclear to me how to do this, becuase there are only two blocks of code in the script. The first isif args.do_train:, where the model is trained, and the second is if args.do_eval:, where the model is evaluated.

I don't want to train, since I'm using a pre-trained model, but I also don't want to evaluate, since my data don't have labels, which makes my use case different than the example of applying the pretrained scibert models to the scierc dataset.

Wondering if you have pointers on how to do this?

Thanks!