Multi Task Vision and Language

Meta Research

Last update: Jan 8, 2023

Related tags

Text Data & NLP vilbert-multi-task

Overview

12-in-1: Multi-Task Vision and Language Representation Learning

Please cite the following if you use this code. Code and pre-trained models for 12-in-1: Multi-Task Vision and Language Representation Learning:

@InProceedings{Lu_2020_CVPR,
author = {Lu, Jiasen and Goswami, Vedanuj and Rohrbach, Marcus and Parikh, Devi and Lee, Stefan},
title = {12-in-1: Multi-Task Vision and Language Representation Learning},
booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

and ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks:

@inproceedings{lu2019vilbert,
  title={Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks},
  author={Lu, Jiasen and Batra, Dhruv and Parikh, Devi and Lee, Stefan},
  booktitle={Advances in Neural Information Processing Systems},
  pages={13--23},
  year={2019}
}

Repository Setup

Create a fresh conda environment, and install all dependencies.

conda create -n vilbert-mt python=3.6
conda activate vilbert-mt
git clone --recursive https://github.com/facebookresearch/vilbert-multi-task.git
cd vilbert-multi-task
pip install -r requirements.txt

Install pytorch

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

Install apex, follows https://github.com/NVIDIA/apex
Install this codebase as a package in this environment.

python setup.py develop

Data Setup

Check README.md under data for more details.

Visiolinguistic Pre-training and Multi Task Training

Pretraining on Conceptual Captions

python train_concap.py --bert_model bert-base-uncased --config_file config/bert_base_6layer_6conect.json --train_batch_size 512 --objective 1 --file_path <path_to_extracted_cc_features>

Download link

Multi-task Training

python train_tasks.py --bert_model bert-base-uncased --from_pretrained <pretrained_model_path> --config_file config/bert_base_6layer_6conect.json --tasks 1-2-4-7-8-9-10-11-12-13-15-17 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model

Download link

Fine-tune from Multi-task trained model

python train_tasks.py --bert_model bert-base-uncased --from_pretrained <multi_task_model_path> --config_file config/bert_base_6layer_6conect.json --tasks 1 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name finetune_from_multi_task_model

License

vilbert-multi-task is licensed under MIT license available in LICENSE file.

Comments

Could you please provide access to the required data files?

Hi! Thank you for releasing this great project! However, I notice that the data files (including the lmdb feature files as well as other metadata) needed to run pre-training and multi-task fine-tuning is not accessible. Could you please add accessible links to them? Or a readme guiding how to generate them is also fine. Thank you very much!

opened by yangapku 10
training_feat_all.lmdb and caption_train.json not found

Hi, thanks for your excellent work.

I'm trying to run your code with Pretraining on Conceptual Captions, but some files do not exist, and I didn't find any script to generate them. I've tried to set up data as specified here, but the step 3 only gives me a data.mdb and a lock.mdb. Can you provide training_feat_all.lmdb and caption_train.json ?

Many thanks.

opened by HubHop 9

i had "AttributeError: 'NoneType' object has no attribute 'named_parameters'" error!

hi i want to train on VCRdataset.

i got this error! how do i do? help T^T

(vilbert-mt) ailab@ailab:~/vilbert-multi-task$ python train_tasks.py --bert_model bert-base-uncased --from_pretrained save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin --config_file config/bert_base_6layer_6conect.json --tasks 5-6 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model
Failed to import tensorflow.
03/12/2020 23:24:28 - INFO - __main__ -   device: cuda n_gpu: 2, distributed training: False, 16-bits training: False
03/12/2020 23:24:29 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/ailab/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
03/12/2020 23:24:29 - INFO - vilbert.task_utils -   Loading VCR_Q-A Dataset with batch size 64
03/12/2020 23:24:40 - INFO - vilbert.task_utils -   Loading VCR_QA-R Dataset with batch size 64
03/12/2020 23:24:53 - INFO - vilbert.utils -   logging file at: save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-multi_task_model/logs
03/12/2020 23:24:53 - ERROR - vilbert.utils -   Model name 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, roberta-base, roberta-large, roberta-large-mnli). We assumed 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
  File "train_tasks.py", line 670, in <module>
    main()
  File "train_tasks.py", line 398, in main
    for key, value in dict(model.named_parameters()).items():
AttributeError: 'NoneType' object has no attribute 'named_parameters'
(vilbert-mt) ailab@ailab:~/vilbert-multi-task$ python train_tasks.py --bert_model bert-base-uncased --from_pretrained save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin --config_file config/bert_base_6layer_6conect.json --tasks 5-6 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model
Failed to import tensorflow.
03/12/2020 23:34:06 - INFO - __main__ -   device: cuda n_gpu: 2, distributed training: False, 16-bits training: False
03/12/2020 23:34:08 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/ailab/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
03/12/2020 23:34:08 - INFO - vilbert.task_utils -   Loading VCR_Q-A Dataset with batch size 64
03/12/2020 23:34:19 - INFO - vilbert.task_utils -   Loading VCR_QA-R Dataset with batch size 64
03/12/2020 23:34:34 - INFO - vilbert.utils -   logging file at: save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-multi_task_model/logs
03/12/2020 23:34:34 - ERROR - vilbert.utils -   Model name 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, roberta-base, roberta-large, roberta-large-mnli). We assumed 'save/bert_base_6_layer_6_connect_freeze_0/pytorch_model_8.bin' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
  File "train_tasks.py", line 670, in <module>
    main()
  File "train_tasks.py", line 398, in main
    for key, value in dict(model.named_parameters()).items():
AttributeError: 'NoneType' object has no attribute 'named_parameters'

opened by jaeyun95 6

Provided Vilbert model Not working with the data

Hi there, I tried to run some demo using the code from this repo. I prepared the data following the instructions in the data folder I.E. download datasets.tar.gz and resnext152 coco features(by wget https://dl.fbaipublicfiles.com/vilbert-multi-task/datasets/coco/features_100/COCO_trainval_resnext152_faster_rcnn_genome.lmdb/data.mdb && mv data.mdb COCO_trainval_resnext152_faster_rcnn_genome.lmdb/)

I tried to evaluate the VQA performance using CUDA_VISIBLE_DEVICES=1 python eval_tasks.py --bert_model bert-base-uncased --from_pretrained multi_task_model.bin --config_file config/bert_base_6layer_6conect.json --tasks 1 --task_specific_tokens

the multi_task_model.bin is download from https://dl.fbaipublicfiles.com/vilbert-multi-task/multi_task_model.bin

But I got

I am wondering if someone else facing the same issues or is there something wrong with my code

opened by jialinw7 4
Where can I find visual genome's object categories?

Hi!

I'm trying to extract features on Conceptual Captions with script/extract_features.py, while the output .npy file contains also the class predictions, I wonder could you also provide the indices corresponding to those 1,600 object categories in Visual Genome? (such as this or this)

Thanks!

opened by juliuswang0728 3
Trying to run demo

I am trying to run demo file. But I can not find the config file (save/resnext_models/e2e_faster_rcnn_X-152-32x8d-FPN_1x_MLP_2048_FPN_512_train.yaml ) and model file (save/resnext_models/model_final.pth). Can anyone help me? Thank you so much.

opened by tranvansanghust 3
errors when running fine tuning cmd

I download your dataset and run: python train_tasks.py --bert_model bert-base-uncased --from_pretrained ./multi_task_model.bin --config_file config/bert_base_6layer_6conect.json --tasks 1 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name finetune_from_multi_task_model

it will give error: Traceback (most recent call last): File "train_tasks.py", line 670, in main() File "train_tasks.py", line 529, in main task_losses, File "/dccstor/extrastore/vilbert-multi-task/vilbert/task_utils.py", line 197, in ForwardModelsTrain batch ValueError: too many values to unpack (expected 9)

opened by YupengGao 3

i got "ConnectionResetError"

hi i run this code for VCR task.(training) i got an error like this.

(vilbert-mt) ailab@ailab:~/vilbert-multi-task$ python train_tasks.py --bert_model bert-base-uncased --from_pretrained save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-pretrained/pytorch_model_19.bin --config_file config/bert_base_6layer_6conect.json --tasks 5-6 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model
Failed to import tensorflow.
03/13/2020 13:52:59 - INFO - __main__ -   device: cuda n_gpu: 2, distributed training: False, 16-bits training: False
03/13/2020 13:53:00 - INFO - pytorch_transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/ailab/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
03/13/2020 13:53:00 - INFO - vilbert.task_utils -   Loading VCR_Q-A Dataset with batch size 16
03/13/2020 13:53:11 - INFO - vilbert.task_utils -   Loading VCR_QA-R Dataset with batch size 16
03/13/2020 13:53:25 - INFO - vilbert.utils -   logging file at: save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-multi_task_model/logs
03/13/2020 13:53:25 - INFO - vilbert.utils -   loading weights file save/VCR_Q-A-VCR_QA-R_bert_base_6layer_6conect-pretrained/pytorch_model_19.bin
03/13/2020 13:53:29 - INFO - vilbert.utils -   Weights of VILBertForVLTasks not initialized from pretrained model: ['bert.embeddings.task_embeddings.weight', 'vil_prediction.logit_fc.0.weight', 'vil_prediction.logit_fc.0.bias', 'vil_prediction.logit_fc.2.weight', 'vil_prediction.logit_fc.2.bias', 'vil_prediction.logit_fc.3.weight', 'vil_prediction.logit_fc.3.bias', 'vil_prediction_gqa.logit_fc.0.weight', 'vil_prediction_gqa.logit_fc.0.bias', 'vil_prediction_gqa.logit_fc.2.weight', 'vil_prediction_gqa.logit_fc.2.bias', 'vil_prediction_gqa.logit_fc.3.weight', 'vil_prediction_gqa.logit_fc.3.bias', 'vil_binary_prediction.logit_fc.0.weight', 'vil_binary_prediction.logit_fc.0.bias', 'vil_binary_prediction.logit_fc.2.weight', 'vil_binary_prediction.logit_fc.2.bias', 'vil_binary_prediction.logit_fc.3.weight', 'vil_binary_prediction.logit_fc.3.bias', 'vil_tri_prediction.weight', 'vil_tri_prediction.bias']
03/13/2020 13:53:29 - INFO - vilbert.utils -   Weights from pretrained model not used in VILBertForVLTasks: ['vil_prediction.main.0.bias', 'vil_prediction.main.0.weight_g', 'vil_prediction.main.0.weight_v', 'vil_prediction.main.3.bias', 'vil_prediction.main.3.weight_g', 'vil_prediction.main.3.weight_v']
559 559
***** Running training *****
  Num Iters:  {'TASK5': 2039, 'TASK6': 2039}
  Batch size:  {'TASK5': 16, 'TASK6': 16}
  Num steps: 40780
Epoch:   0%|                                             | 0/20 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "train_tasks.py", line 670, in <module>
    main()
  File "train_tasks.py", line 529, in main
    task_losses,
  File "/home/ailab/vilbert-multi-task/vilbert/task_utils.py", line 200, in ForwardModelsTrain
    if task_cfg[task_id]["process"] in ["dialog"]:
KeyError: 'process'
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/site-packages/torch/utils/data/_utils/pin_memory.py", line 21, in _pin_memory_loop
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 276, in rebuild_storage_fd
    fd = df.detach()
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 493, in Client
    answer_challenge(c, authkey)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 737, in answer_challenge
    response = connection.recv_bytes(256)        # reject large message
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/ailab/anaconda3/envs/vilbert-mt/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

how do i do? help T^T

opened by jaeyun95 2

Bug in Validation Logger

Hi,

In the line below we aren't multiplying the score with batch-size, and while logging we are dividing the sum of scores with the complete dataset-size. Does this seem correct?

https://github.com/facebookresearch/vilbert-multi-task/blob/a290ba6d9c1c95375f622e66bfcdb666dd01ba89/vilbert/utils.py#L271

https://github.com/facebookresearch/vilbert-multi-task/blob/a290ba6d9c1c95375f622e66bfcdb666dd01ba89/vilbert/utils.py#L339

opened by yashkant 2
Bump tensorflow from 2.5.1 to 2.5.2
Bumps tensorflow from 2.5.1 to 2.5.2.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.5.2

Release 2.5.2

This release introduces several vulnerability fixes:

Fixes a code injection issue in saved_model_cli (CVE-2021-41228)

Fixes a vulnerability due to use of uninitialized value in Tensorflow (CVE-2021-41225)

Fixes a heap OOB in FusedBatchNorm kernels (CVE-2021-41223)

Fixes an arbitrary memory read in ImmutableConst (CVE-2021-41227)

Fixes a heap OOB in SparseBinCount (CVE-2021-41226)

Fixes a heap OOB in SparseFillEmptyRows (CVE-2021-41224)

Fixes a segfault due to negative splits in SplitV (CVE-2021-41222)

Fixes segfaults and vulnerabilities caused by accesses to invalid memory during shape inference in Cudnn* ops (CVE-2021-41221)

Fixes a null pointer exception when Exit node is not preceded by Enter op (CVE-2021-41217)

Fixes an integer division by 0 in tf.raw_ops.AllToAll (CVE-2021-41218)

Fixes an undefined behavior via nullptr reference binding in sparse matrix multiplication (CVE-2021-41219)

Fixes a heap buffer overflow in Transpose (CVE-2021-41216)

Prevents deadlocks arising from mutually recursive tf.function objects (CVE-2021-41213)

Fixes a null pointer exception in DeserializeSparse (CVE-2021-41215)

Fixes an undefined behavior arising from reference binding to nullptr in tf.ragged.cross (CVE-2021-41214)

Fixes a heap OOB read in tf.ragged.cross (CVE-2021-41212)

Fixes a heap OOB read in all tf.raw_ops.QuantizeAndDequantizeV* ops (CVE-2021-41205)

Fixes an FPE in ParallelConcat (CVE-2021-41207)

Fixes FPE issues in convolutions with zero size filters (CVE-2021-41209)

Fixes a heap OOB read in tf.raw_ops.SparseCountSparseOutput (CVE-2021-41210)

Fixes vulnerabilities caused by incomplete validation in boosted trees code (CVE-2021-41208)

Fixes vulnerabilities caused by incomplete validation of shapes in multiple TF ops (CVE-2021-41206)

Fixes a segfault produced while copying constant resource tensor (CVE-2021-41204)

Fixes a vulnerability caused by unitialized access in EinsumHelper::ParseEquation (CVE-2021-41201)

Fixes several vulnerabilities and segfaults caused by missing validation during checkpoint loading (CVE-2021-41203)

Fixes an overflow producing a crash in tf.range (CVE-2021-41202)

Fixes an overflow producing a crash in tf.image.resize when size is large (CVE-2021-41199)

Fixes an overflow producing a crash in tf.tile when tiling tensor is large (CVE-2021-41198)

Fixes a vulnerability produced due to incomplete validation in tf.summary.create_file_writer (CVE-2021-41200)

Fixes multiple crashes due to overflow and CHECK-fail in ops with large tensor shapes (CVE-2021-41197)

Fixes a crash in max_pool3d when size argument is 0 or negative (CVE-2021-41196)

Fixes a crash in tf.math.segment_* operations (CVE-2021-41195)

Updates curl to 7.78.0 to handle CVE-2021-22922, CVE-2021-22923, CVE-2021-22924, CVE-2021-22925, and CVE-2021-22926.

Changelog

Sourced from tensorflow's changelog.

Release 2.5.2

This release introduces several vulnerability fixes:

Fixes a code injection issue in saved_model_cli (CVE-2021-41228)

Fixes a vulnerability due to use of uninitialized value in Tensorflow (CVE-2021-41225)

Fixes a heap OOB in FusedBatchNorm kernels (CVE-2021-41223)

Fixes an arbitrary memory read in ImmutableConst (CVE-2021-41227)

Fixes a heap OOB in SparseBinCount (CVE-2021-41226)

Fixes a heap OOB in SparseFillEmptyRows (CVE-2021-41224)

Fixes a segfault due to negative splits in SplitV (CVE-2021-41222)

Fixes segfaults and vulnerabilities caused by accesses to invalid memory during shape inference in Cudnn* ops (CVE-2021-41221)

Fixes a null pointer exception when Exit node is not preceded by Enter op (CVE-2021-41217)

Fixes an integer division by 0 in tf.raw_ops.AllToAll (CVE-2021-41218)

Fixes an undefined behavior via nullptr reference binding in sparse matrix multiplication (CVE-2021-41219)

Fixes a heap buffer overflow in Transpose (CVE-2021-41216)

Prevents deadlocks arising from mutually recursive tf.function objects (CVE-2021-41213)

Fixes a null pointer exception in DeserializeSparse (CVE-2021-41215)

Fixes an undefined behavior arising from reference binding to nullptr in tf.ragged.cross (CVE-2021-41214)

Fixes a heap OOB read in tf.ragged.cross (CVE-2021-41212)

Fixes a heap OOB read in all tf.raw_ops.QuantizeAndDequantizeV* ops (CVE-2021-41205)

Fixes an FPE in ParallelConcat ([CVE-2021-41207] (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41207))

Fixes FPE issues in convolutions with zero size filters (CVE-2021-41209)

Fixes a heap OOB read in tf.raw_ops.SparseCountSparseOutput (CVE-2021-41210)

Fixes vulnerabilities caused by incomplete validation in boosted trees code (CVE-2021-41208)

Fixes vulnerabilities caused by incomplete validation of shapes in multiple TF ops (CVE-2021-41206)

... (truncated)

Commits

957590e Merge pull request #52873 from tensorflow-jenkins/relnotes-2.5.2-20787

2e1d16d Update RELEASE.md

2fa6dd9 Merge pull request #52877 from tensorflow-jenkins/version-numbers-2.5.2-192

4807489 Merge pull request #52881 from tensorflow/fix-build-1-on-r2.5

d398bdf Disable failing test

857ad5e Merge pull request #52878 from tensorflow/fix-build-1-on-r2.5

6c2a215 Disable failing test

f5c57d4 Update version numbers to 2.5.2

e51f949 Insert release notes place-fill

2620d2c Merge pull request #52863 from tensorflow/fix-build-3-on-r2.5

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies
opened by dependabot[bot] 1
VQA and GQA test splits do not have answers?
Hi, I notice that for VQA and GQA, if we want to test the pretrained model, there are no available ground-truth answers.

if "test" not in self.split: answer = entry["answer"] labels = answer["labels"] scores = answer["scores"] if labels is not None: target.scatter_(0, labels, scores)

I wonder how you calculate the score on the test set?
opened by ranery 1
Bump numpy from 1.16.4 to 1.21.0
Bumps numpy from 1.16.4 to 1.21.0.

Release notes

Sourced from numpy's releases.

v1.21.0

NumPy 1.21.0 Release Notes

The NumPy 1.21.0 release highlights are

continued SIMD work covering more functions and platforms,

initial work on the new dtype infrastructure and casting,

universal2 wheels for Python 3.8 and Python 3.9 on Mac,

improved documentation,

improved annotations,

new PCG64DXSM bitgenerator for random numbers.

In addition there are the usual large number of bug fixes and other improvements.

The Python versions supported for this release are 3.7-3.9. Official support for Python 3.10 will be added when it is released.

:warning: Warning: there are unresolved problems compiling NumPy 1.21.0 with gcc-11.1 .

Optimization level -O3 results in many wrong warnings when running the tests.

On some hardware NumPy will hang in an infinite loop.

New functions

Add PCG64DXSM BitGenerator

Uses of the PCG64 BitGenerator in a massively-parallel context have been shown to have statistical weaknesses that were not apparent at the first release in numpy 1.17. Most users will never observe this weakness and are safe to continue to use PCG64. We have introduced a new PCG64DXSM BitGenerator that will eventually become the new default BitGenerator implementation used by default_rng in future releases. PCG64DXSM solves the statistical weakness while preserving the performance and the features of PCG64.

See upgrading-pcg64 for more details.

(gh-18906)

Expired deprecations

The shape argument numpy.unravel_index cannot be passed as dims keyword argument anymore. (Was deprecated in NumPy 1.16.)

... (truncated)

Commits

b235f9e Merge pull request #19283 from charris/prepare-1.21.0-release

34aebc2 MAINT: Update 1.21.0-notes.rst

493b64b MAINT: Update 1.21.0-changelog.rst

07d7e72 MAINT: Remove accidentally created directory.

032fca5 Merge pull request #19280 from charris/backport-19277

7d25b81 BUG: Fix refcount leak in ResultType

fa5754e BUG: Add missing DECREF in new path

61127bb Merge pull request #19268 from charris/backport-19264

143d45f Merge pull request #19269 from charris/backport-19228

d80e473 BUG: Removed typing for == and != in dtypes

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies
opened by dependabot[bot] 0
Multi-Task Training Needs 50 Days?

Hi, thanks for providing such an impressive codebase!

I am trying to train the multi-task model, it seems that 1k iterations cost around 30 hours, so if the total iteration is 40k, then the training time would be 40 x 30 / 24 = 50 days!!

Is that true on your side? Do you have any recipe to accelerate the training process?

Best regards, Haoran

opened by ranery 0
Received signal 7, Begin Mangled Stack Trace

I am trying to run the Multi-Task Training with the given command but it seems to be giving me an error. *** Received signal 7 *** *** BEGIN MANGLED STACK TRACE *** *** Received signal 11 *** *** BEGIN MANGLED STACK TRACE ***

Furthermore, I am getting another error ERROR: Unexpected segmentation fault encountered in worker. when trying to run the Fine-tune from Multi-task trained model using the multi_task_model.bin model.

Are these two issues related? How can one solve them?

opened by Ceralu 0

TypeError: Couldn't build proto file into descriptor pool

I fixed numpy version error in requirements.txt.

Now I want to execute this command according to this paragraph in README.md:

python train_tasks.py --bert_model bert-base-uncased --from_pretrained {path to}/multi_task_model.bin --config_file config/bert_base_6layer_6conect.json --tasks 1-2-4-7-8-9-10-11-12-13-15-17 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model

I encountered a error, this command's output is:

2021-06-25 11:22:37.444926: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "train_tasks.py", line 33, in <module>
    from vilbert.task_utils import (
  File "/LocalRun/home/shihao.wang/vilbert-multi-task/vilbert/task_utils.py", line 19, in <module>
    from vilbert.datasets import DatasetMapTrain, DatasetMapEval
  File "/LocalRun/home/shihao.wang/vilbert-multi-task/vilbert/datasets/__init__.py", line 6, in <module>
    from .concept_cap_dataset import (
  File "/LocalRun/home/shihao.wang/vilbert-multi-task/vilbert/datasets/concept_cap_dataset.py", line 15, in <module>
    import tensorpack.dataflow as td
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorpack/__init__.py", line 5, in <module>
    from tensorpack.libinfo import __version__, __git_version__, _HAS_TF
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorpack/libinfo.py", line 53, in <module>
    import tensorflow as tf  # noqa
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/__init__.py", line 55, in <module>
    from ._api.v2 import compat
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/__init__.py", line 39, in <module>
    from . import v1
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v1/__init__.py", line 34, in <module>
    from . import compat
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v1/compat/__init__.py", line 40, in <module>
    from . import v2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v1/compat/v2/__init__.py", line 32, in <module>
    from tensorflow._api.v2.compat.v2 import __internal__
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v2/__init__.py", line 37, in <module>
    from . import compat
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v2/compat/__init__.py", line 40, in <module>
    from . import v2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorflow/_api/v2/compat/v2/compat/v2/__init__.py", line 331, in <module>
    from tensorboard.summary._tf import summary
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/summary/__init__.py", line 22, in <module>
    from tensorboard.summary import v1  # noqa: F401
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/summary/v1.py", line 21, in <module>
    from tensorboard.plugins.audio import summary as _audio_summary
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/plugins/audio/summary.py", line 34, in <module>
    from tensorboard.plugins.audio import metadata
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/plugins/audio/metadata.py", line 18, in <module>
    from tensorboard.compat.proto import summary_pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/summary_pb2.py", line 17, in <module>
    from tensorboard.compat.proto import tensor_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/tensor_pb2.py", line 16, in <module>
    from tensorboard.compat.proto import resource_handle_pb2 as tensorboard_dot_compat_dot_proto_dot_resource__handle__pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/resource_handle_pb2.py", line 16, in <module>
    from tensorboard.compat.proto import tensor_shape_pb2 as tensorboard_dot_compat_dot_proto_dot_tensor__shape__pb2
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/tensorboard/compat/proto/tensor_shape_pb2.py", line 23, in <module>
    serialized_pb=_b('\n+tensorboard/compat/proto/tensor_shape.proto\x12\x0btensorboard\"{\n\x10TensorShapeProto\x12.\n\x03\x64im\x18\x02 \x03(\x0b\x32!.tensorboard.TensorShapeProto.Dim\x12\x14\n\x0cunknown_rank\x18\x03 \x01(\x08\x1a!\n\x03\x44im\x12\x0c\n\x04size\x18\x01 \x01(\x03\x12\x0c\n\x04name\x18\x02 \x01(\tB\x87\x01\n\x18org.tensorflow.frameworkB\x11TensorShapeProtosP\x01ZSgithub.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_shape_go_proto\xf8\x01\x01\x62\x06proto3')
  File "/LocalRun/miniconda3/envs/vilbert-mt/lib/python3.6/site-packages/google/protobuf/descriptor.py", line 983, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "tensorboard/compat/proto/tensor_shape.proto":
  tensorboard.TensorShapeProto.dim: "tensorboard.TensorShapeProto.dim" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.unknown_rank: "tensorboard.TensorShapeProto.unknown_rank" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.Dim.size: "tensorboard.TensorShapeProto.Dim.size" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.Dim.name: "tensorboard.TensorShapeProto.Dim.name" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.Dim: "tensorboard.TensorShapeProto.Dim" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto: "tensorboard.TensorShapeProto" is already defined in file "tensorboardX/src/tensor_shape.proto".
  tensorboard.TensorShapeProto.dim: "tensorboard.TensorShapeProto.Dim" seems to be defined in "tensorboardX/src/tensor_shape.proto", which is not imported by "tensorboard/compat/proto/tensor_shape.proto".  To use it here, please add the necessary import.

opened by zhonhel 2

Multi Task Vision and Language

Related tags

Overview

12-in-1: Multi-Task Vision and Language Representation Learning

Repository Setup

Data Setup

Visiolinguistic Pre-training and Multi Task Training

Pretraining on Conceptual Captions

Multi-task Training

Fine-tune from Multi-task trained model

License

Comments

TensorFlow 2.5.2

Release 2.5.2

Release 2.5.2

v1.21.0

NumPy 1.21.0 Release Notes

New functions

Add PCG64DXSM BitGenerator

Expired deprecations

Owner

Meta Research

Using Bert as the backbone model for lime, designed for NLP task explanation (sentence pair text classification task)

A Neural Language Style Transfer framework to transfer natural language text smoothly between fine-grained language styles like formal/casual, active/passive, and many more. Created by Prithiviraj Damodaran. Open to pull requests and other forms of collaboration.

PhoNLP: A BERT-based multi-task learning toolkit for part-of-speech tagging, named entity recognition and dependency parsing

A simple recipe for training and inferencing Transformer architecture for Multi-Task Learning on custom datasets. You can find two approaches for achieving this in this repo.

A Survey of Natural Language Generation in Task-Oriented Dialogue System (TOD): Recent Advances and New Frontiers

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Meta learning algorithms to train cross-lingual NLI (multi-task) models

Code for EmBERT, a transformer model for embodied, language-guided visual task completion.

Ptorch NLU, a Chinese text classification and sequence annotation toolkit, supports multi class and multi label classification tasks of Chinese long text and short text, and supports sequence annotation tasks such as Chinese named entity recognition, part of speech tagging and word segmentation.

Code of paper: A Recurrent Vision-and-Language BERT for Navigation

Pytorch code for ICRA'21 paper: "Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation"

History Aware Multimodal Transformer for Vision-and-Language Navigation

PyTorch implementation of "data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language" from Meta AI

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

Code for paper Multitask-Finetuning of Zero-shot Vision-Language Models

Simple Python script to scrape youtube channles of "Parity Technologies and Web3 Foundation" and translate them to well-known braille language or any language

multi-label，classifier，text classification，多标签文本分类，文本分类，BERT，ALBERT，multi-label-classification，seq2seq，attention，beam search

Indobenchmark are collections of Natural Language Understanding (IndoNLU) and Natural Language Generation (IndoNLG)

Partially offline multi-language translator built upon Huggingface transformers.