I directly run the script following README, but obtained abnormal results for NYT Exact Match setting, with the F1 = 0.667201.

My command is python -m main --bert_directory ../pretrained_model/bert_base_uncased_huggingface/ --num_generated_triplets 15 --max_grad_norm 1 --na_rel_coef 0.5 --max_epoch 100 --max_span_length 10

Here is some excerpted logs which might be attributed to:

• My config

Unparsed args: ['--num_generated_triplets', '15'] dataset_name : NYT-exact train_file : ./data/NYT/exact_data/train.json valid_file : ./data/NYT/exact_data/valid.json test_file : ./data/NYT/exact_data/test.json generated_data_directory : ./data/generated_data/ generated_param_directory : ./data/generated_data/model_param/ bert_directory : ../pretrained_model/bert_base_uncased_huggingface/ partial : False model_name : Set-Prediction-Networks num_generated_triples : 10 num_decoder_layers : 3 matcher : avg na_rel_coef : 0.5 rel_loss_weight : 1 head_ent_loss_weight : 2 tail_ent_loss_weight : 2 fix_bert_embeddings : True batch_size : 8 max_epoch : 100 gradient_accumulation_steps : 1 decoder_lr : 2e-05 encoder_lr : 1e-05 lr_decay : 0.01 weight_decay : 1e-05 max_grad_norm : 1.0 optimizer : AdamW n_best_size : 100 max_span_length : 10 refresh : False use_gpu : True visible_gpu : 1 random_seed : 1 DATA SUMMARY START: Relation Alphabet Size: 24 Train Instance Number: 56196 Valid Instance Number: 5000 Test Instance Number: 5000 DATA SUMMARY END. Data setting is saved to file: ./data/generated_data/NYT-exact_Set-Prediction-Networks_data.pickle

• And the results:

=== Epoch 99 train === Instance: 800; loss: 0.0072 Instance: 1600; loss: 0.0061 Instance: 2400; loss: 0.0053 Instance: 3200; loss: 0.0054 Instance: 4000; loss: 0.0055 Instance: 4800; loss: 0.0049 Instance: 5600; loss: 0.0045 Instance: 6400; loss: 0.0046 Instance: 7200; loss: 0.0042 Instance: 8000; loss: 0.0041 Instance: 8800; loss: 0.0044 Instance: 9600; loss: 0.0043 Instance: 10400; loss: 0.0046 Instance: 11200; loss: 0.0046 Instance: 12000; loss: 0.0046 Instance: 12800; loss: 0.0043 Instance: 13600; loss: 0.0042 Instance: 14400; loss: 0.0040 Instance: 15200; loss: 0.0040 Instance: 16000; loss: 0.0039 Instance: 16800; loss: 0.0038 Instance: 17600; loss: 0.0037 Instance: 18400; loss: 0.0036 Instance: 19200; loss: 0.0035 Instance: 20000; loss: 0.0035 Instance: 20800; loss: 0.0035 Instance: 21600; loss: 0.0034 Instance: 22400; loss: 0.0034 Instance: 23200; loss: 0.0033 Instance: 24000; loss: 0.0034 Instance: 24800; loss: 0.0033 Instance: 25600; loss: 0.0033 Instance: 26400; loss: 0.0033 Instance: 27200; loss: 0.0034 Instance: 28000; loss: 0.0034 Instance: 28800; loss: 0.0034 Instance: 29600; loss: 0.0033 Instance: 30400; loss: 0.0035 Instance: 31200; loss: 0.0038 Instance: 32000; loss: 0.0038 Instance: 32800; loss: 0.0038 Instance: 33600; loss: 0.0037 Instance: 34400; loss: 0.0039 Instance: 35200; loss: 0.0039 Instance: 36000; loss: 0.0039 Instance: 36800; loss: 0.0038 Instance: 37600; loss: 0.0038 Instance: 38400; loss: 0.0039 Instance: 39200; loss: 0.0039 Instance: 40000; loss: 0.0039 Instance: 40800; loss: 0.0039 Instance: 41600; loss: 0.0038 Instance: 42400; loss: 0.0038 Instance: 43200; loss: 0.0038 Instance: 44000; loss: 0.0039 Instance: 44800; loss: 0.0039 Instance: 45600; loss: 0.0041 Instance: 46400; loss: 0.0041 Instance: 47200; loss: 0.0040 Instance: 48000; loss: 0.0040 Instance: 48800; loss: 0.0039 Instance: 49600; loss: 0.0039 Instance: 50400; loss: 0.0039 Instance: 51200; loss: 0.0039 Instance: 52000; loss: 0.0039 Instance: 52800; loss: 0.0039 Instance: 53600; loss: 0.0039 Instance: 54400; loss: 0.0039 Instance: 55200; loss: 0.0039 Instance: 56000; loss: 0.0039 === Epoch 99 Test === ------Num of Gold Triplet is 1------ gold_num = 3240 pred_num = 3732 right_num = 2409 relation_right_num = 2826 entity_right_num = 2864 precision = 0.6454983922829582 recall = 0.7435185185185185 f1_value = 0.6910499139414802 rel_precision = 0.7572347266881029 rel_recall = 0.8722222222222222 rel_f1_value = 0.810671256454389 ent_precision = 0.767416934619507 ent_recall = 0.8839506172839506 ent_f1_value = 0.8215720022948939 ------Num of Gold Triplet is 2------ gold_num = 2094 pred_num = 1675 right_num = 1309 relation_right_num = 1425 entity_right_num = 1476 precision = 0.7814925373134328 recall = 0.625119388729704 f1_value = 0.6946139559564872 rel_precision = 0.8507462686567164 rel_recall = 0.6805157593123209 rel_f1_value = 0.7561687450252057 ent_precision = 0.8811940298507462 ent_recall = 0.7048710601719198 ent_f1_value = 0.7832316264261078 ------Num of Gold Triplet is 3------ gold_num = 942 pred_num = 590 right_num = 501 relation_right_num = 537 entity_right_num = 542 precision = 0.8491525423728814 recall = 0.5318471337579618 f1_value = 0.6540469973890339 rel_precision = 0.9101694915254237 rel_recall = 0.5700636942675159 rel_f1_value = 0.7010443864229765 ent_precision = 0.9186440677966101 ent_recall = 0.5753715498938429 ent_f1_value = 0.7075718015665795 ------Num of Gold Triplet is 4------ gold_num = 1160 pred_num = 619 right_num = 530 relation_right_num = 576 entity_right_num = 559 precision = 0.8562197092084006 recall = 0.45689655172413796 f1_value = 0.59584035975267 rel_precision = 0.9305331179321487 rel_recall = 0.496551724137931 rel_f1_value = 0.6475548060708264 ent_precision = 0.9030694668820679 ent_recall = 0.4818965517241379 ent_f1_value = 0.628442945474986 ------Num of Gold Triplet is greater than or equal to 5------ gold_num = 684 pred_num = 266 right_num = 241 relation_right_num = 254 entity_right_num = 251 precision = 0.9060150375939849 recall = 0.35233918128654973 f1_value = 0.5073684210526316 rel_precision = 0.9548872180451128 rel_recall = 0.3713450292397661 rel_f1_value = 0.5347368421052632 ent_precision = 0.943609022556391 ent_recall = 0.3669590643274854 ent_f1_value = 0.5284210526315789 ------Normal Triplets------ gold_num = 2028 pred_num = 2259 right_num = 1414 relation_right_num = 1725 entity_right_num = 1604 precision = 0.6259406817175741 recall = 0.6972386587771203 f1_value = 0.6596687660368556 rel_precision = 0.7636122177954847 rel_recall = 0.8505917159763313 rel_f1_value = 0.8047585724282714 ent_precision = 0.7100486941124391 ent_recall = 0.7909270216962525 ent_f1_value = 0.748308840681129 ------Multiply label Triplets------ gold_num = 4079 pred_num = 2511 right_num = 2052 relation_right_num = 2202 entity_right_num = 2255 precision = 0.8172043010752689 recall = 0.5030644765873988 f1_value = 0.6227617602427922 rel_precision = 0.8769414575866189 rel_recall = 0.5398381956361853 rel_f1_value = 0.6682852807283762 ent_precision = 0.8980485862206292 ent_recall = 0.5528315763667565 ent_f1_value = 0.684370257966616 ------Overlapping Triplets------ gold_num = 5530 pred_num = 4243 right_num = 3319 relation_right_num = 3608 entity_right_num = 3791 precision = 0.7822295545604525 recall = 0.6001808318264015 f1_value = 0.6792182543742965 rel_precision = 0.8503417393353759 rel_recall = 0.6524412296564195 rel_f1_value = 0.7383607899314436 ent_precision = 0.8934716002828188 ent_recall = 0.6855334538878842 ent_f1_value = 0.7758109076025785 gold_num = 8120 pred_num = 6882 right_num = 4990 relation_right_num = 5618 entity_right_num = 5692 precision = 0.7250799186283057 recall = 0.6145320197044335 f1_value = 0.6652446340487934 rel_precision = 0.8163324614937518 rel_recall = 0.691871921182266 rel_f1_value = 0.7489668044260765 ent_precision = 0.8270851496657948 ent_recall = 0.7009852216748769 ent_f1_value = 0.7588321557125718 Best result on test set is 0.667201 achieving at epoch 90. /pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of add_ is deprecated: add_(Number alpha, Tensor other) Consider using one of the following signatures instead: add_(Tensor other, *, Number alpha)

Right now I'm guessing two possible reasons: 1.I see that the README has just updated recently, so I have not set --num_generated_triplets correctly, but I'm not sure this would lead to dramaticly performance drop from expected 90 to 66. 2.The script and model default to set fix_bert_embeddings=True, as can be refered to here, which is not the usual case where people finetune BERT on downstream tasks instead of freeze it.

Great thanks for your attention and help!

举个例子： pred: [(0, 1, 2, 3, 4), (2, 1, 2,3,4)] gold: [(0, 1, 2, 3, 4)]

按照代码里面的结果：

rel_recall = 1/1 = 100% ent_recall = 2/1 = 200% (因为ent_num = 2)

这样会导致，当gold有很多组（假设N）， 而pred只预测对一组实体，但是这组实体对应了不同关系，所以重复了多次（假设M），最后也会被认为ent_num+=1，真实召回应该是1/N，按照代码计算可能是 M/N

您好，感谢分享。有一个问题，论文table 4的实验结果： 我根据您提供源代码和命令行，效果只到0.928841, 请问，可以分享更详细的参数，或者训练过程用了哪些技巧么 ？感谢您的回复！ python -m main --bert_directory BERT_DIR --batch_size 4 --num_generated_triplets 10 --na_rel_coef 0.25 --max_grad_norm 20 --max_epoch 100 --encoder_lr 0.00002 --decoder_lr 0.00005 --num_decoder_layers 4 --max_span_length 10 --weight_decay 0.000001 --lr_decay 0.02

=== Epoch 73 Test === precision = 0.930599, recall = 0.927090, f1_value = 0.928841 rel_precision = 0.955205, rel_recall = 0.951603, rel_f1_value = 0.953401 ent_precision = 0.950158, ent_recall = 0.946574, ent_f1_value = 0.948363 Achieving Best Result on Test Set

我按论文中写的设置，并不能得到论文中的结果……请问是存在什么问题吗？ webNLG_partial

Best result on test set is 0.930512 achieving at epoch 8
python -m main --bert_directory /data/home/wuyuming/wxl/pretrained_models/bert-base-cased \
--batch_size 4 \
--num_generated_triples 10 \
--na_rel_coef 0.25 \
--max_grad_norm 20 \
--max_epoch 100 \
--encoder_lr 0.00002 \
--decoder_lr 0.00005 \
--num_decoder_layers 3 \
--max_span_length 10 \
--weight_decay 0.000001 \
--lr_decay 0.02 \
--refresh True \
--visible_gpu 1

webNLG_exact: 同样设置下：Best result on test set is 0.837710 achieving at epoch 98. nyt_exact:在epoch7达到 0.8536778597111341后，recall迅速下降，f1变为0.1…… nyt_partial:在epoch8达到0.8662156396659961后，recall迅速下降，f1变为0.1……

ython -m main --bert_directory /data/home/wuyuming/wxl/pretrained_models/bert-base-cased \
--batch_size 4 \
--num_generated_triples 10 \
--na_rel_coef 0.25 \
--max_grad_norm 20 \
--max_epoch 100 \
--encoder_lr 0.00002 \
--decoder_lr 0.00005 \
--num_decoder_layers 3 \
--max_span_length 10 \
--weight_decay 0.000001 \
--lr_decay 0.02 \
--refresh True \
--visible_gpu 0 \
--train_file ./data/NYT/casrel_data/new_train.json \
--valid_file ./data/NYT/casrel_data/new_valid.json \
--test_file ./data/NYT/casrel_data/new_test.json

我使用您readme里的WebNLG partial match启动命令，得到结果是Best result on test set is 0.928166 achieving at epoch 64，这和论文中的93.4有一点距离，请问93.4是不是您得到的最好结果，而不是平均结果呢？ 另外在metric.py中似乎都是exact matching，因为他要求了实体得头尾都相同，如果我把代码改成partial matching 是否可能进一步达到93.4呢