Hello,

I am executing the training script, and got the following output. I am following the instructions exactly as described in the READ.ME.

(tvlnbert) rehmanm@kw60805:/data1/vln-bert$ python -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 --node_rank=0 train.py --from_pretrained data/models/vilbert_pytorch_model_9.bin --save_name pre_train_run_id --num_epochs 50 --warmup_proportion 0.08 --cooldown_factor 8 --masked_language --masked_vision --no_ranking

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

02/15/2021 18:43:27 - INFO - pytorch_pretrained_bert.tokenization - loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/rehmanm/.pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084 02/15/2021 18:43:27 - INFO - pytorch_pretrained_bert.tokenization - loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/rehmanm/.pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084 02/15/2021 18:43:27 - INFO - main - using provided training trajectories 02/15/2021 18:44:49 - WARNING - utils.dataset.beam_dataset - skipping index: 603 in beam data in from path: data/beamsearch/beams_val_seen.json 02/15/2021 18:44:49 - WARNING - utils.dataset.beam_dataset - skipping index: 661 in beam data in from path: data/beamsearch/beams_val_seen.json 02/15/2021 18:44:49 - WARNING - utils.dataset.beam_dataset - skipping index: 1016 in beam data in from path: data/beamsearch/beams_val_seen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 32 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 76 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 121 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 268 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 285 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 298 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 301 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 306 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 361 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 384 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 439 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 487 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 537 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 547 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 791 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 813 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 909 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 918 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 929 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 974 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1128 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1204 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1245 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1281 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1312 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1391 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1430 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1587 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1651 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1769 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1775 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1857 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1860 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1906 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1928 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 1996 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2061 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2070 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2085 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2128 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2204 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2226 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2235 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2261 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - WARNING - utils.dataset.beam_dataset - skipping index: 2329 in beam data in from path: data/beamsearch/beams_val_unseen.json 02/15/2021 18:44:57 - INFO - main - batch_size: 4 02/15/2021 18:44:57 - INFO - vilbert.vilbert - loading archive file data/models/vilbert_pytorch_model_9.bin 02/15/2021 18:44:57 - INFO - vilbert.vilbert - Model config { "attention_probs_dropout_prob": 0.1, "bi_attention_type": 1, "bi_hidden_size": 1024, "bi_intermediate_size": 1024, "bi_num_attention_heads": 8, "fast_mode": false, "fixed_t_layer": 0, "fixed_v_layer": 0, "fusion_method": "mul", "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "in_batch_pairs": false, "initializer_range": 0.02, "intermediate_size": 3072, "intra_gate": false, "max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12, "pooling_method": "mul", "predict_feature": false, "t_biattention_id": [ 6, 7, 8, 9, 10, 11 ], "type_vocab_size": 2, "v_attention_probs_dropout_prob": 0.1, "v_biattention_id": [ 0, 1, 2, 3, 4, 5 ], "v_feature_size": 2048, "v_hidden_act": "gelu", "v_hidden_dropout_prob": 0.1, "v_hidden_size": 1024, "v_initializer_range": 0.02, "v_intermediate_size": 1024, "v_num_attention_heads": 8, "v_num_hidden_layers": 6, "v_target_size": 1601, "vocab_size": 30522, "with_coattention": true }

02/15/2021 18:45:02 - INFO - vilbert.vilbert - Weights of VLNBert not initialized from pretrained model: ['bert.v_embeddings.image_orientation_embeddings.weight', 'bert.v_embeddings.image_orientation_embeddings.bias', 'bert.v_embeddings.image_next_orientation_embeddings.weight', 'bert.v_embeddings.image_next_orientation_embeddings.bias', 'bert.v_embeddings.image_sequence_embeddings.weight', 'vil_logit.weight', 'vil_logit.bias'] 02/15/2021 18:45:02 - INFO - main - number of parameters: 250,086,014 02/15/2021 18:45:04 - INFO - main - using distributed data parallel 02/15/2021 18:45:04 - INFO - main - starting training... /home/rehmanm/anaconda3/envs/tvlnbert/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:231: UserWarning: To get the last learning rate computed by the scheduler, please use get_last_lr(). warnings.warn("To get the last learning rate computed by the scheduler, " /data1/vln-bert/vilbert/optimization.py:166: UserWarning: This overload of add_ is deprecated: add_(Number alpha, Tensor other) Consider using one of the following signatures instead: add_(Tensor other, *, Number alpha) (Triggered internally at /opt/conda/conda-bld/pytorch_1595629411241/work/torch/csrc/utils/python_arg_parser.cpp:766.) exp_avg.mul_(beta1).add_(1.0 - beta1, grad) /data1/vln-bert/vilbert/optimization.py:166: UserWarning: This overload of add_ is deprecated: add_(Number alpha, Tensor other) Consider using one of the following signatures instead: add_(Tensor other, *, Number alpha) (Triggered internally at /opt/conda/conda-bld/pytorch_1595629411241/work/torch/csrc/utils/python_arg_parser.cpp:766.) exp_avg.mul_(beta1).add_(1.0 - beta1, grad)

I would be grateful if you could let me know if this is expected or if I need to change something to fix this error.

Hi Arjun,

I have some issues with reproducing your weights. I run the exact same commands as you wrote in your README.md. However, the performances are less good than yours.

Here's the tensorboard of my success rate on val seen and val unseen when finetuning with ranking loss:

One explanation could that the pre-training phase was not trained for enough time. On the following plots, it sounds as is 50 epochs is not enough.

Also, I wasn't testing with the exact same dependencies version, which might explain the differences. I am testing again with it.

Thanks for your guidance, Pierre

Hi Arjun,

I am trying to reproduce your results for your speaker + follower + VLNBERT baseline from your paper.

Can you confirm me that what you did is basically train VLNBERT with the augmented paths from Tan et al.: http://www.cs.unc.edu/~airsplay/aug_paths.json ?

Have you done any pre-processing step such that joining the augmented paths to create longer paths?

Do you mind sharing the beams and the weights of this baseline?

Thanks, Pierre

Hi,

Do you plan to share the precomputed dataset with features extracted from bottomup attention?

The manual computation is quite heavy to setup.

Thanks for your answer!

The output test.py is producing a score file + a result file.

In order to get proper metrics, I guess the result file should be fed into the eval.py from the original R2R codebase.

But do you have a version that is directly compatible with your codebase?

Thanks!

The output test.py is producing a score file + a result file.

In order to get proper metrics, I guess the result file should be fed into the eval.py from the original R2R codebase.

But do you have a version that is directly compatible with your codebase?

Thanks!

When loading the model with weights 1+2+3 in eval.py (I am using only a subset of the precomputed dataset), I receive the warning:

11/01/2020 14:53:40 - INFO - vilbert.vilbert -   Weights from pretrained model not used in VLNBert: ['vil_prediction.logit_fc.0.weight', 'vil_prediction.logit_fc.0.bias', 'vil_prediction.logit_fc.2.weight', 'vil_prediction.logit_fc.2.bias', 'vil_prediction.logit_fc.3.weight', 'vil_prediction.logit_fc.3.bias', 'vision_logit.weight', 'vision_logit.bias', 'linguisic_logit.weight', 'linguisic_logit.bias']

Apparently, the model should be VILBERTForVLTasks whereas it is loading VLNBert. Is that correct?

Hi, I am trying to reproduce the work from the scratch. I used it on 24GB titan rtx and it is out of memory. May I ask what resources are required? Like how many GPU hours and what type of GPU.

Hi Arjun,

Thanks a lot for releasing the code! Is it possible that you can also share the code to reproduce the results in Table 3 of the paper? For example the way of combining with speaker follower models. Thanks in advance!

Hi Arjun,

Thanks again for sharing your code ;)

The INSTALL.md is referring to download-auxilary-data.py. I guess you didn't have time yet to upload it.

Please tell me when you can :)

In your INSTALL.md, data preprocessing is summarized as a python script to download existing processed files.

However, as your VLN-BERT model is transferable, I would like to try this on other datasets and other models. Hence, how to pre-process the data is very important to me.

To be specific, I would like to learn about the settings of how to generate beam_[dataset].json and the scores_[dataset].json and which model is used to compute those beams (Is it just following this github repo? Have you ever changed anything?). As if we would like to combine VLN-BERT with Follower and Speaker, we have to use data from beam_[dataset].json.

Could you please release more details about that?

Thanks a lot!

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Hi there, I could not find a license file in the repo. I see that Vilbert is under MIT license, are you also using MIT license? Could you please add a license file to the repo?