Hey WadeYin9712, absolutely love the work. During fine-tuning sstphrase, I can across the above error. I was looking for the error when I found you had used it in the Input features. While working with BERT, I haven't come across this masking. Does this have to do with the new module pytorch_pretrained_bert.modeling_new? Because that was also an error that I had to change to pytorch_pretrained_bert.modeling to import BertforPretraining. Could you help me out with this? Thanks in advance!

During pretraining, the epoch was killed. I assumed it was the RAM consumption, so i tried to increase swap memory. I tried it again but it seems like it is taking a really long time, even after connecting to the GPU. The result shown:

CUDA_VISIBLE_DEVICES=0 python3 finetune_on_pregenerated_sstphrase.py \

--pregenerated_data /home/oem/Documents/SentiBERT/training_sstphrase
--bert_model bert-base-uncased
--do_lower_case
--output_dir /home/oem/Documents/SentiBERT/results/sstphrase_pretrain
--epochs 3 2020-09-11 21:05:33,356: device: cpu n_gpu: 0, distributed training: False, 16-bits training: False 2020-09-11 21:05:34,533: loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/oem/.cache/torch/pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084 2020-09-11 21:05:37,707: loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at /home/oem/.cache/torch/pytorch_pretrained_bert/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157 2020-09-11 21:05:37,707: loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at /home/oem/.cache/torch/pytorch_pretrained_bert/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517 2020-09-11 21:05:37,708: Model config { "architectures": [ "BertForMaskedLM" ], "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-12, "max_position_embeddings": 512, "model_type": "bert", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 0, "type_vocab_size": 2, "vocab_size": 30522 }

2020-09-11 21:05:47,259: Weights of BertForPreTraining not initialized from pretrained model: ['bert.pooler_phrase.dense.weight', 'bert.pooler_phrase.dense.bias', 'bert.trans.weight', 'bert.trans.bias', 'bert.trans_2.weight', 'bert.trans_2.bias', 'bert.trans_3.weight', 'bert.trans_3.bias', 'bert.bahdanau_attention.linear_encoder.weight', 'bert.bahdanau_attention.linear_encoder.bias', 'bert.bahdanau_attention.linear_decoder.weight', 'bert.bahdanau_attention.linear_decoder.bias', 'bert.bahdanau_attention.linear_in.0.weight', 'bert.bahdanau_attention.linear_in.0.bias', 'bert.bahdanau_attention.linear_in.3.weight', 'bert.bahdanau_attention.linear_in.3.bias', 'bert.bahdanau_attention_3.linear_encoder.weight', 'bert.bahdanau_attention_3.linear_encoder.bias', 'bert.bahdanau_attention_3.linear_decoder.weight', 'bert.bahdanau_attention_3.linear_decoder.bias', 'bert.bahdanau_attention_3.linear_in.0.weight', 'bert.bahdanau_attention_3.linear_in.0.bias', 'bert.bahdanau_attention_3.linear_in.3.weight', 'bert.bahdanau_attention_3.linear_in.3.bias', 'bert.linear_out.0.weight', 'bert.linear_out.0.bias', 'bert.linear_out.3.weight', 'bert.linear_out.3.bias', 'bert.linear_out_3.0.weight', 'bert.linear_out_3.0.bias', 'bert.linear_out_3.3.weight', 'bert.linear_out_3.3.bias', 'bert.LayerNorm.weight', 'bert.LayerNorm.bias', 'cls.phrase_predictions.transform.dense.weight', 'cls.phrase_predictions.transform.dense.bias', 'cls.phrase_predictions.transform.LayerNorm.weight', 'cls.phrase_predictions.transform.LayerNorm.bias', 'cls.phrase_predictions.decoder.weight', 'cls.phrase_predictions.decoder.bias'] 2020-09-11 21:05:47,267: ***** Running training ***** 2020-09-11 21:05:47,267: Num examples = 25137 2020-09-11 21:05:47,267: Batch size = 32 2020-09-11 21:05:47,268: Num steps = 785 2020-09-11 21:05:47,539: Loading training examples for epoch 0 Training examples: 100%|██████████████████| 8379/8379 [01:00<00:00, 139.13it/s] 2020-09-11 21:06:56,683: Loading complete! Epoch 0: 0%| | 1/262 [18:56<82:19:17, 1135.47s/it, Loss: 4.97729]

Estimated time for each epoch is roughly 80+ hours. I am using a Gtx 1050ti with a nvidia 440 driver, 8GB RAM and 16GB of swap memory. The nvidia-smi is listed below:

nvidia-smi Fri Sep 11 19:21:59 2020
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX 105... Off | 00000000:01:00.0 Off | N/A | | N/A 52C P0 N/A / N/A | 148MiB / 4042MiB | 0% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 992 G /usr/lib/xorg/Xorg 64MiB | | 0 1446 G /usr/bin/gnome-shell 83MiB | +-----------------------------------------------------------------------------+

Could you let me now how to improve timing? In a previoud issue, you had mentioned you were able to achieve minutes for each epoch.

Hi, thanks for the nice work. I tried to run Sentibert finetuning with emocontext:

CUDA_VISIBLE_DEVICES=0 python examples/run_classifier_new.py
--task_name emocontext
--do_train
--do_eval
--do_lower_case
--data_dir datasets/emocontext
--pretrain_dir results/sstphrase_pretrain/
--bert_model bert-base-uncased
--max_seq_length 128
--train_batch_size 16
--learning_rate 0.001
--num_train_epochs 10
--output_dir results/emocontext
--seed 1
--para sentibert

But ran into the following error: ERROR - pytorch_pretrained_bert.modeling_new - Model name 'results/sstphrase_pretrain/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc). We assumed 'results/sstphrase_pretrain/sstphrase.json' was a path or url but couldn't find any file associated to this path or url.

It looks like sstphrase.json is missing. I only managed to obtain sstphrase.bin from your download link. Can you please also provide the .json config file? Thanks.

Hey,

I looked into the preprocessing python file and you have used a user defined pytorch_pretrained_bert.tokenization instead of using the huggingface library transformer.

Will i get the same result for sentiBERT if I use the below code:

from transformers import BertTokenizer

If not, could you tell me what is the difference between them?

Thanks!

Hey, thank you for the work! While training sst2, i came upon the FileNotFountError: [Errno 2] No such file or directory: '/stanford-corenlp-full-2018-10-05/sst2_train_text.txt.json'. Although, this json was created in the previous step and stored in the same directory i.e., in the stanford-corenlp-full-2018-10-05 folder. Any assistance would be of great help! Thank you!

Hello, I followed all the process as written in the GitHub page. But I got only 51% accuracy for sst-2 task. Then I tried checkpoints offered by your teams and didn't train by myself, which means I only do preprocessing and evaluation, and the results are still about 50%. And my colleague met the same problem. Do anyone met the same condition? What could possible be the problem?

And if its preprocessing is different from SentiBERT? If different, could it be possible for you to release, too?

Also, I wonder how SentiBERT compares to the sentiment parse of stanford-corenlp.

Could it be possible for you to release the model for SentiBERT w/ token?

Thanks for your time :)

Hi @WadeYin9712 and thanks so much for sharing your code. Would you please give some instruction how to do sentiment classification on a new dataset?(positive/negative/neutral) I want to use your pretreated model on the datasets to do classification on a totally new dataset.

Thanks for your time :)

Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.phrase_predictions.transform.dense.weight', 'cls.phrase_predictions.transform.dense.bias', 'cls.phrase_predictions.transform.LayerNorm.weight', 'cls.phrase_predictions.transform.LayerNorm.bias', 'cls.phrase_predictions.decoder.weight', 'cls.phrase_predictions.decoder.bias'] Are these weights essential during finetuning? Did you come across the same problem when implementing sentibert? If so, how did you solve it? The bin file and json file are downloaed from your link. "这些参数没有加载成功是否会影响整体的准确率呀，您在实现的时候这些参数加载成功了么,模型文件和json文件是从您提供的链接下载的，应该没有问题"

Thanks very much for your work. I am quite interested in your local/global difficulty analysis. Wonder would it be possible for you to release the corresponding code you used to make the split (how to calculate switch number) . We tried to split according to your description by our own but could not be able to produce the exact number.....(for ours, we manage to get a maximum global switches of 43 if consider leaves and 14 if only at phrase level while you report a maximum of 23 in the paper)

Thanks in advance Best

Is there an updated version? Instead of getting the *span_3.npy and *span.npy files from the SST generated json, emoint_st.py created a span_3.npy and split.npy file. The two span.npy files are needed for run_classifier_dataset_utils_new.npy.

I am very interested in your program, but I encountered a problem in the program and processing part, how to divide this, and where do you put the two paths? Did not see that your file exists