Hey I am wondering if it is possible to release the original pretrained model with some parameters of LM head prediction, such as linear layers, instead of the current version (drop those parameters off).

Hi, I maintain FitBERT, a library that uses BERT to fill in the blanks (do masked language modeling). An example:

from fitbert import FitBert


# in theory you can pass a model_name and tokenizer
# currently supported models: bert-large-uncased and distilbert-base-uncased
# this takes a while and loads a whole big BERT into memory
fb = FitBert()

masked_string = "Why Bert, you're looking ***mask*** today!"
options = ['buff', 'handsome', 'strong']

ranked_options = fb.rank(masked_string, options=options)
# >>> ['handsome', 'strong', 'buff']

I think that SpanBERT should be an improvement to using BERT when the mask covers more than 1 token. However, I am not seeing SBO anywhere in this code. Is it not released? Should I just mask sequential tokens and hope that the model is better at predicting them?

Also, since you don't do NSP, does the [SEP] tag have any meaning? If not, is there no special EOS token?

Thanks so much!

I am using the same python packages as specified in requirements.txt and README (apex@4a8c4ac). Python version 3.6.10 I get the following error after running the SQuAD 2.0 command about 10 minutes in.

05/06/2020 08:17:34 - INFO - pytorch_pretrained_bert.modeling - Weights of BertForQuestionAnswering not initialized from pretrained model: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias', 'qa_outputs.weight', 'qa_outputs.bias']
Traceback (most recent call last):
  File "code/run_squad.py", line 1138, in <module>
    main(args)
  File "code/run_squad.py", line 947, in main
    max_grad_norm=1.0)
  File "/home/user/.conda3/envs/spanbert/lib/python3.6/site-packages/apex-0.1-py3.6.egg/apex/optimizers/fused_adam.py", line 40, in __init__
  File "/home/user/.conda3/envs/spanbert/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'fused_adam_cuda'

I'm confused about how SpanBERT fit the training set.

I replaced this line: https://github.com/facebookresearch/SpanBERT/blob/master/code/run_tacred.py#L337

eval_examples = processor.get_dev_examples(args.data_dir)

with

eval_examples = processor.get_train_examples(args.data_dir)

The training F1 is very low

accuracy = 0.8149991192531266 eval_loss = 1.294708618564385 f1 = 0.09820465025017167 n_correct = 1001 n_gold = 13012 n_pred = 7374 num = 68124 precision = 0.13574721996202874 recall = 0.07692898862588379

I just wanted to check if you are planning on merging this with the transformers library since that would make it immensely easier to use. For now, just extracting the archive and calling the Bert APIs has worked for me but the solution feels inelegant as I have to download the model separately. Surprisingly, loading the BertModel using this codebase and the tokenizer using the transformers library results in random results across runs (with the same batch order and content). Can't really point out where the issue is for now!

I'm trying to convert the run_tacred.ty to a Huggingface's Transformers compatible version. I load the SpanBERT fine-tuned model for tacred and tokenizer using the following code:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("SpanBERT/spanbert-base-cased")
model = AutoModel.from_pretrained("SpanBERT/spanbert-base-cased")

But at the evaluation time, when I get to the following section:

with torch.no_grad():
     logits = model(input_ids, segment_ids, input_mask)
loss_fct = CrossEntropyLoss()
tmp_eval_loss = loss_fct(logits.view(-1, num_labels), label_ids.view(-1))

I get the following error:

AttributeError: 'tuple' object has no attribute 'view'

which means the output of the model, logits, is not a torch tensor. I'm trying to figure out what the correct size of logits is for the loss_fct function. logits has two elements where each of them is torch.Size([32, 128, 1024]) where 32 is the batch size. label_ids here is also torch.Size([32]).

Based on the dimensions, I tried the following for the loss_fct based on the acceptable input dimension for loss_fct:

tmp_eval_loss = loss_fct(logits[0].view(list(input_ids.size())[0], -1), label_ids.view(-1))

which even though passes the computing loss function step, it throws an error in compute_f1 since we'll have the preds of size (n, 1024) where n is the total number of data points we are evaluating while the labels's size is n.

I wonder how should I set the dimensions?

Hi, thanks for the great work!

I'm trying to replicate the relation extraction results on TACRED but haven't succeeded yet. Currently I'm unable to achieve an F1 score above 67.2, so I want to clarify a few things.

NVIDIA Apex

The first issue (which applies to all tasks) regards NVIDIA Apex. Due to Apex transitioning to the new amp API, commit #512 moved FP16_Optimizer and FusedAdam to the apex.contrib module. Users that compiled Apex from the latest master and want to use SpanBERT's --fp16 get an exception "Please install apex from [...]", because the import of FP16_Optimizer and FusedAdam fails (e.g. in run_tacred.py).

As a temporary solution, I resorted to an earlier commit (#424) of Apex without these breaking changes. Can you specify the commit you used to compile Apex?

Fine-tuning

For fine-tuning on TACRED, I used the following configuration (based on the README and what I gathered from the paper).

python code/run_tacred.py \
  --do_train \
  --do_eval \
  --data_dir <TACRED_DATA_DIR> \
  --model spanbert-base-cased \
  --train_batch_size 32 \
  --eval_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 10 \
  --max_seq_length 128 \
  --output_dir tacred_dir \
  --fp16

Namespace object written to the training logs:

Namespace(data_dir='../relex-data/tacred/', do_eval=True, do_lower_case=False, do_train=True, eval_batch_size=32, eval_metric='f1', eval_per_epoch=10, eval_test=False, feature_mode='ner', fp16=True, gradient_accumulation_steps=1, learning_rate=2e-05, loss_scale=0, max_seq_length=128, model='spanbert-base-cased', negative_label='no_relation', no_cuda=False, num_train_epochs=10.0, output_dir='model_save', seed=42, train_batch_size=32, train_mode='random_sorted', warmup_proportion=0.1)

eval_results.txt

accuracy = 0.8535636958154743
batch_size = 32
epoch = 8
eval_loss = 0.7800797500179313
f1 = 0.6727970716900277
global_step = 18728
learning_rate = 2e-05
precision = 0.6535993061578491
recall = 0.6931567328918322

Test set results (test_results.txt, after running run_tacred.py with eval_test=True):

accuracy = 0.8534311342848305
eval_loss = 0.7800683651940298
f1 = 0.6725
precision = 0.6533657182512145
recall = 0.6927888153053716

• Can you confirm that this is the same configuration used to produce the results stated in the paper?
• Are there plans to make the fine-tuned TACRED model publicly available?

A typo at https://github.com/facebookresearch/SpanBERT/blob/10641ea3795771dd96e9e3e9ef0ead4f4f6a29d2/code/pytorch_pretrained_bert/modeling.py#L551 will make people confused when loading model from_tf checkpoint.

And fixed it according to this line https://github.com/facebookresearch/SpanBERT/blob/10641ea3795771dd96e9e3e9ef0ead4f4f6a29d2/code/pytorch_pretrained_bert/modeling.py#L50

I downloaded the spanbert_hf_base.tar file and when I load it torch.load(spanbert_hf_base.tar), it gave me the error KeyError: "filename 'storages' not found". How do I properly load the pretrained model? Thanks

I tried to train using run_mrqa.py with the json files (by changing the read files codes). However, I am unable to proceed further. Any help would be appreciated. Please share the preparation for HotpotQA dataset using SpanBERT.

Training SQuAD 2.0 with the command given in README.md raises following error

Traceback (most recent call last):
  File "code/run_squad.py", line 1138, in <module>
    main(args)
  File "code/run_squad.py", line 947, in main
    max_grad_norm=1.0)
TypeError: __init__() got an unexpected keyword argument 'max_grad_norm'

According to https://nvidia.github.io/apex/optimizers.html FusedAdam doesn't take a max_grad_norm argument.

Bumps certifi from 2019.9.11 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I am using spanbert model ( I do not fine-tune it) downloaded from https://github.com/facebookresearch/SpanBERT to test it on some text samples.

The result I get is in this format using HF:

token_classifier = pipeline("token-classification", model=model, aggregation_strategy="max",tokenizer=tokenizer,grouped_entities=True)

Label_0, Label_1 How to map these Labels to entity names? Is there any function, documents where it mentions about entity type mapping?

​ ​

Bumps pillow from 6.1.0 to 9.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I have trained a pretrained bert model, but I feel that many weights are not well trained. As a result, the output of different input texts is almost similar, and the difference may only be 7 decimal places behind. What is the cause

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

I want to train spanBert from scratch, for contextual span embedding. The architecture is quite large and I think only three layers will suffice for my task. Please help me with how can I reduce the size of the architecture and train it on the objective of spanbert. Thanks