Hi! Thank you for releasing this great project! However, I notice that the data files (including the lmdb feature files as well as other metadata) needed to run pre-training and multi-task fine-tuning is not accessible. Could you please add accessible links to them? Or a readme guiding how to generate them is also fine. Thank you very much!

Hi, thanks for your excellent work.

I'm trying to run your code with Pretraining on Conceptual Captions, but some files do not exist, and I didn't find any script to generate them. I've tried to set up data as specified here, but the step 3 only gives me a data.mdb and a lock.mdb. Can you provide training_feat_all.lmdb and caption_train.json ?

Many thanks.

hi i want to train on VCRdataset.

i got this error! how do i do? help T^T

(vilbert-mt) ailab@ailab:~/vilbert-multi-task$ python train_tasks.py --bert_model bert-base-uncased --from_pretrained save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin --config_file config/bert_base_6layer_6conect.json --tasks 5-6 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model
Failed to import tensorflow.
03/12/2020 23:24:28 - INFO - __main__ -   device: cuda n_gpu: 2, distributed training: False, 16-bits training: False
03/12/2020 23:24:29 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/ailab/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
03/12/2020 23:24:29 - INFO - vilbert.task_utils -   Loading VCR_Q-A Dataset with batch size 64
03/12/2020 23:24:40 - INFO - vilbert.task_utils -   Loading VCR_QA-R Dataset with batch size 64
03/12/2020 23:24:53 - INFO - vilbert.utils -   logging file at: save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-multi_task_model/logs
03/12/2020 23:24:53 - ERROR - vilbert.utils -   Model name 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, roberta-base, roberta-large, roberta-large-mnli). We assumed 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
  File "train_tasks.py", line 670, in <module>
    main()
  File "train_tasks.py", line 398, in main
    for key, value in dict(model.named_parameters()).items():
AttributeError: 'NoneType' object has no attribute 'named_parameters'
(vilbert-mt) ailab@ailab:~/vilbert-multi-task$ python train_tasks.py --bert_model bert-base-uncased --from_pretrained save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin --config_file config/bert_base_6layer_6conect.json --tasks 5-6 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model
Failed to import tensorflow.
03/12/2020 23:34:06 - INFO - __main__ -   device: cuda n_gpu: 2, distributed training: False, 16-bits training: False
03/12/2020 23:34:08 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/ailab/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
03/12/2020 23:34:08 - INFO - vilbert.task_utils -   Loading VCR_Q-A Dataset with batch size 64
03/12/2020 23:34:19 - INFO - vilbert.task_utils -   Loading VCR_QA-R Dataset with batch size 64
03/12/2020 23:34:34 - INFO - vilbert.utils -   logging file at: save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-multi_task_model/logs
03/12/2020 23:34:34 - ERROR - vilbert.utils -   Model name 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, roberta-base, roberta-large, roberta-large-mnli). We assumed 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
  File "train_tasks.py", line 670, in <module>
    main()
  File "train_tasks.py", line 398, in main
    for key, value in dict(model.named_parameters()).items():
AttributeError: 'NoneType' object has no attribute 'named_parameters'

Hi there, I tried to run some demo using the code from this repo. I prepared the data following the instructions in the data folder I.E. download datasets.tar.gz and resnext152 coco features(by wget https://dl.fbaipublicfiles.com/vilbert-multi-task/datasets/coco/features_100/COCO_trainval_resnext152_faster_rcnn_genome.lmdb/data.mdb && mv data.mdb COCO_trainval_resnext152_faster_rcnn_genome.lmdb/)

I tried to evaluate the VQA performance using CUDA_VISIBLE_DEVICES=1 python eval_tasks.py --bert_model bert-base-uncased --from_pretrained multi_task_model.bin --config_file config/bert_base_6layer_6conect.json --tasks 1 --task_specific_tokens

the multi_task_model.bin is download from https://dl.fbaipublicfiles.com/vilbert-multi-task/multi_task_model.bin

But I got

I am wondering if someone else facing the same issues or is there something wrong with my code

Hi!

I'm trying to extract features on Conceptual Captions with script/extract_features.py, while the output .npy file contains also the class predictions, I wonder could you also provide the indices corresponding to those 1,600 object categories in Visual Genome? (such as this or this)

Thanks!

I am trying to run demo file. But I can not find the config file (save/resnext_models/e2e_faster_rcnn_X-152-32x8d-FPN_1x_MLP_2048_FPN_512_train.yaml ) and model file (save/resnext_models/model_final.pth). Can anyone help me? Thank you so much.

I download your dataset and run: python train_tasks.py --bert_model bert-base-uncased --from_pretrained ./multi_task_model.bin --config_file config/bert_base_6layer_6conect.json --tasks 1 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name finetune_from_multi_task_model

it will give error: Traceback (most recent call last): File "train_tasks.py", line 670, in main() File "train_tasks.py", line 529, in main task_losses, File "/dccstor/extrastore/vilbert-multi-task/vilbert/task_utils.py", line 197, in ForwardModelsTrain batch ValueError: too many values to unpack (expected 9)

hi i run this code for VCR task.(training) i got an error like this.

(vilbert-mt) ailab@ailab:~/vilbert-multi-task$ python train_tasks.py --bert_model bert-base-uncased --from_pretrained save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-pretrained/pytorch_model_19.bin --config_file config/bert_base_6layer_6conect.json --tasks 5-6 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model
Failed to import tensorflow.
03/13/2020 13:52:59 - INFO - __main__ -   device: cuda n_gpu: 2, distributed training: False, 16-bits training: False
03/13/2020 13:53:00 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/ailab/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
03/13/2020 13:53:00 - INFO - vilbert.task_utils -   Loading VCR_Q-A Dataset with batch size 16
03/13/2020 13:53:11 - INFO - vilbert.task_utils -   Loading VCR_QA-R Dataset with batch size 16
03/13/2020 13:53:25 - INFO - vilbert.utils -   logging file at: save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-multi_task_model/logs
03/13/2020 13:53:25 - INFO - vilbert.utils -   loading weights file save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-pretrained/pytorch_model_19.bin
03/13/2020 13:53:29 - INFO - vilbert.utils -   Weights of VILBertForVLTasks not initialized from pretrained model: ['bert.embeddings.task_embeddings.weight', 'vil_prediction.logit_fc.0.weight', 'vil_prediction.logit_fc.0.bias', 'vil_prediction.logit_fc.2.weight', 'vil_prediction.logit_fc.2.bias', 'vil_prediction.logit_fc.3.weight', 'vil_prediction.logit_fc.3.bias', 'vil_prediction_gqa.logit_fc.0.weight', 'vil_prediction_gqa.logit_fc.0.bias', 'vil_prediction_gqa.logit_fc.2.weight', 'vil_prediction_gqa.logit_fc.2.bias', 'vil_prediction_gqa.logit_fc.3.weight', 'vil_prediction_gqa.logit_fc.3.bias', 'vil_binary_prediction.logit_fc.0.weight', 'vil_binary_prediction.logit_fc.0.bias', 'vil_binary_prediction.logit_fc.2.weight', 'vil_binary_prediction.logit_fc.2.bias', 'vil_binary_prediction.logit_fc.3.weight', 'vil_binary_prediction.logit_fc.3.bias', 'vil_tri_prediction.weight', 'vil_tri_prediction.bias']
03/13/2020 13:53:29 - INFO - vilbert.utils -   Weights from pretrained model not used in VILBertForVLTasks: ['vil_prediction.main.0.bias', 'vil_prediction.main.0.weight_g', 'vil_prediction.main.0.weight_v', 'vil_prediction.main.3.bias', 'vil_prediction.main.3.weight_g', 'vil_prediction.main.3.weight_v']
559 559
***** Running training *****
  Num Iters:  {'TASK5': 2039, 'TASK6': 2039}
  Batch size:  {'TASK5': 16, 'TASK6': 16}
  Num steps: 40780
Epoch:   0%|                                             | 0/20 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "train_tasks.py", line 670, in <module>
    main()
  File "train_tasks.py", line 529, in main
    task_losses,
  File "/home/ailab/vilbert-multi-task/vilbert/task_utils.py", line 200, in ForwardModelsTrain
    if task_cfg[task_id]["process"] in ["dialog"]:
KeyError: 'process'
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/site-packages/torch/utils/data/_utils/pin_memory.py", line 21, in _pin_memory_loop
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 276, in rebuild_storage_fd
    fd = df.detach()
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 493, in Client
    answer_challenge(c, authkey)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 737, in answer_challenge
    response = connection.recv_bytes(256)        # reject large message
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

how do i do? help T^T

Hi,

In the line below we aren't multiplying the score with batch-size, and while logging we are dividing the sum of scores with the complete dataset-size. Does this seem correct?

https://github.com/facebookresearch/vilbert-multi-task/blob/a290ba6d9c1c95375f622e66bfcdb666dd01ba89/vilbert/utils.py#L271

https://github.com/facebookresearch/vilbert-multi-task/blob/a290ba6d9c1c95375f622e66bfcdb666dd01ba89/vilbert/utils.py#L339

Bumps tensorflow from 2.5.1 to 2.5.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hi, I notice that for VQA and GQA, if we want to test the pretrained model, there are no available ground-truth answers.

if "test" not in self.split:
      answer = entry["answer"]
      labels = answer["labels"]
      scores = answer["scores"]
      if labels is not None:
          target.scatter_(0, labels, scores)

I wonder how you calculate the score on the test set?

Bumps numpy from 1.16.4 to 1.21.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hi, thanks for providing such an impressive codebase!

I am trying to train the multi-task model, it seems that 1k iterations cost around 30 hours, so if the total iteration is 40k, then the training time would be 40 x 30 / 24 = 50 days!!

Is that true on your side? Do you have any recipe to accelerate the training process?

Best regards, Haoran

I am trying to run the Multi-Task Training with the given command but it seems to be giving me an error. *** Received signal 7 *** *** BEGIN MANGLED STACK TRACE *** *** Received signal 11 *** *** BEGIN MANGLED STACK TRACE ***

Furthermore, I am getting another error ERROR: Unexpected segmentation fault encountered in worker. when trying to run the Fine-tune from Multi-task trained model using the multi_task_model.bin model.

Are these two issues related? How can one solve them?

I fixed numpy version error in requirements.txt.

Now I want to execute this command according to this paragraph in README.md:

python train_tasks.py --bert_model bert-base-uncased --from_pretrained {path to}/multi_task_model.bin --config_file config/bert_base_6layer_6conect.json --tasks 1-2-4-7-8-9-10-11-12-13-15-17 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model

I encountered a error, this command's output is:

2021-06-25 11:22:37.444926: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "train_tasks.py", line 33, in <module>
    from vilbert.task_utils import (
  File "/LocalRun/home/shihao.wang/vilbert-multi-task/vilbert/task_utils.py", line 19, in <module>
    from vilbert.datasets import DatasetMapTrain, DatasetMapEval
  File "/LocalRun/home/shihao.wang/vilbert-multi-task/vilbert/datasets/__init__.py", line 6, in <module>
    from .concept_cap_dataset import (
  File "/LocalRun/home/shihao.wang/vilbert-multi-task/vilbert/datasets/concept_cap_dataset.py", line 15, in <module>
    import tensorpack.dataflow as td
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorpack/__init__.py", line 5, in <module>
    from tensorpack.libinfo import __version__, __git_version__, _HAS_TF
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorpack/libinfo.py", line 53, in <module>
    import tensorflow as tf  # noqa
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/__init__.py", line 55, in <module>
    from ._api.v2 import compat
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/__init__.py", line 39, in <module>
    from . import v1
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v1/__init__.py", line 34, in <module>
    from . import compat
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v1/compat/__init__.py", line 40, in <module>
    from . import v2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v1/compat/v2/__init__.py", line 32, in <module>
    from tensorflow._api.v2.compat.v2 import __internal__
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v2/__init__.py", line 37, in <module>
    from . import compat
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v2/compat/__init__.py", line 40, in <module>
    from . import v2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v2/compat/v2/__init__.py", line 331, in <module>
    from tensorboard.summary._tf import summary
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/summary/__init__.py", line 22, in <module>
    from tensorboard.summary import v1  # noqa: F401
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/summary/v1.py", line 21, in <module>
    from tensorboard.plugins.audio import summary as _audio_summary
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/plugins/audio/summary.py", line 34, in <module>
    from tensorboard.plugins.audio import metadata
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/plugins/audio/metadata.py", line 18, in <module>
    from tensorboard.compat.proto import summary_pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/summary_pb2.py", line 17, in <module>
    from tensorboard.compat.proto import tensor_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/tensor_pb2.py", line 16, in <module>
    from tensorboard.compat.proto import resource_handle_pb2 as tensorboard_dot_compat_dot_proto_dot_resource__handle__pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/resource_handle_pb2.py", line 16, in <module>
    from tensorboard.compat.proto import tensor_shape_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__shape__pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/tensor_shape_pb2.py", line 23, in <module>
    serialized_pb=_b('\n+tensorboard/compat/proto/tensor_shape.proto\x12\x0btensorboard\"{\n\x10TensorShapeProto\x12.\n\x03\x64im\x18\x02 \x03(\x0b\x32!.tensorboard.TensorShapeProto.Dim\x12\x14\n\x0cunknown_rank\x18\x03 \x01(\x08\x1a!\n\x03\x44im\x12\x0c\n\x04size\x18\x01 \x01(\x03\x12\x0c\n\x04name\x18\x02 \x01(\tB\x87\x01\n\x18org.tensorflow.frameworkB\x11TensorShapeProtosP\x01ZSgithub.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_shape_go_proto\xf8\x01\x01\x62\x06proto3')
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/google/protobuf/descriptor.py", line 983, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "tensorboard/compat/proto/tensor_shape.proto":
  tensorboard.TensorShapeProto.dim: "tensorboard.TensorShapeProto.dim" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.unknown_rank: "tensorboard.TensorShapeProto.unknown_rank" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.Dim.size: "tensorboard.TensorShapeProto.Dim.size" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.Dim.name: "tensorboard.TensorShapeProto.Dim.name" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.Dim: "tensorboard.TensorShapeProto.Dim" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto: "tensorboard.TensorShapeProto" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.dim: "tensorboard.TensorShapeProto.Dim" seems to be defined in "tensorboardX/src/tensor_shape.proto", which is not imported by "tensorboard/compat/proto/tensor_shape.proto".  To use it here, please add the necessary import.