Hi, thank you for your interesting work! I have a problem when trying to download the provided dataset annotations and features since the one drive link provided cannot be visited in China without a VPN. So it's difficult for me to prepare the data on my ubuntu machine. Do you have any generous advice for me to solve this problem? Or would you please provide another download link that can be easily connected in China for MSCOCO data? Thank you!

HI, I am trying to use one V100 GPU with 16G memory to run the fine-tuning on COCO image captioning task and always encounter such error "ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)". So what is the GPU memory of your V100 and what are the recommended configurations (e.g., batch_size, num_workers) on my cluster with 4*V100 of 16G memory for running fine-tuning for COCO captioning. Thanks!

Hi Luowei --

I'm unable to reproduce the image features that you've published here for COCO and CC. I've trained and evaluated the model using your provided features as well as my extracted features, on the VQA2 task (VQA2 uses COCO images). There is still an outstanding gap in performance. While you report 67.4, I can only achieve 64.3. This is a significant 3-point gap. I am wondering if others have encountered similar problem and how they have resolved it?

I've extracted my own features using the script you shared with me privately (slightly modified to resole dependency issues). Using the housebw/detectron image and your provided detectron checkpoint .pkl and config .yaml, I generate different features than yours. Comparing image-by-image, I have different values in the tensors/matricies. I also get different aggregate statisics (min, max, mean, variance) for features, image-by-image. This is the same situation for CC as well. I've also confirmed it is not a precision issue as well (float16 vs float32).

As it stands, I cannot replicate your results despite my best efforts to follow all your provided documentation, using the same environment, code, data dependencies, and source data.

I am attempting to use your SOTA model on a new dataset/task. Not being able to replicate your results is an impediment...

Thanks, Shawn

Might it be possible to release the script to extract features using the Detectron model? Probably something similar to this https://github.com/ruotianluo/ImageCaptioning.pytorch/blob/master/scripts/prepro_feats.py that was used to generate the region_feat_gvd_wo_bgd/trainval .npz features and class probabilities?

Hi, I just want to test the captioning result on some raw images. I have read the vlp/decode_img2txt.py, but the settings are a little bit complicated for me, for example, the standard size of an input image.

So it would be very kind of you if a simple usage could be provided.

I really appreciate any help you can provide.

Hi Luowei, Recently I have been trying to extract detectron visual features following your guideline here. But I still cannot replicate the features. I use the recommended docker image and refer to the preliminary scripts you send me by email. However, your script extract_features_luowei.py is actually incompatible with the detectron model in the housebw's repo. For example, the im_detect_bbox method (in the figure below) should be imported from core.test instead of core.test_engine.

Actually, I also find several other incompatibility issues when trying to run your code inside the provided docker environment. Even after I fixed some incompatibility issues and run the code successfully, I encounter such errors:

WARNING cnn.py:  40: [====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information.
INFO net.py:  59: Loading weights from: /export/home/vlp_data/e2e_faster_rcnn_X-101-64x4d-FPN_2x.pkl
I1029 08:52:12.747747  2485 net_dag_utils.cc:118] Operator graph pruning prior to chain compute took: 0.000186569 secs
I1029 08:52:12.747993  2485 net_dag.cc:61] Number of parallel execution chains 36 Number of operators = 371
I1029 08:52:12.770298  2485 net_dag_utils.cc:118] Operator graph pruning prior to chain compute took: 0.000165061 secs
I1029 08:52:12.770571  2485 net_dag.cc:61] Number of parallel execution chains 30 Number of operators = 358
/export/home/vlp_data/coco_raw/coco_tiny/COCO_test2014_000000000001.jpg
terminate called after throwing an instance of 'caffe2::EnforceNotMet'
  what():  [enforce fail at blob.h:94] IsType<T>(). wrong type for the Blob instance. Blob contains nullptr (uninitialized) while caller expects caffe2::Tensor<caffe2::CUDAContext> .
Offending Blob name: gpu_0/rois_fpn2.
Error from operator: 
input: "gpu_0/fpn_res2_2_sum" input: "gpu_0/rois_fpn2" output: "gpu_0/roi_feat_fpn2" name: "" type: "RoIAlign" arg { name: "pooled_h" i: 7 } arg { name: "sampling_ratio" i: 2 } arg { name: "spatial_scale" f: 0.25 } arg { name: "pooled_w" i: 7 } device_option { device_type: 1 cuda_gpu_id: 0 }
*** Aborted at 1572339134 (unix time) try "date -d @1572339134" if you are using GNU date ***
terminate called recursively
terminate called recursively
PC: @     0x7fe85f011428 gsignal
*** SIGABRT (@0x9b5) received by PID 2485 (TID 0x7fe7c0ffd700) from PID 2485; stack trace: ***
    @     0x7fe85f3b7390 (unknown)
    @     0x7fe85f011428 gsignal
    @     0x7fe85f01302a abort
    @     0x7fe8590bf84d __gnu_cxx::__verbose_terminate_handler()
terminate called recursively
    @     0x7fe8590bd6b6 (unknown)
    @     0x7fe8590bd701 std::terminate()
    @     0x7fe8590e8d38 (unknown)
    @     0x7fe85f3ad6ba start_thread
    @     0x7fe85f0e341d clone
    @                0x0 (unknown)
Aborted (core dumped)

Due to these incompatibility issues, I find it pretty difficult to extract the same visual features as yours. But if we use other detectron codes like detectron2 or mmdetection, we cannot use your pre-trained models. Therefore, I would like to ask when you can fully release your detectron code (code & python environment), which will be extremely helpful for those planning to apply your VLP model into their own datasets like me. Look forward to your reply:)

The provided fine-tuning scripts fails on multiple GPUs machine. Traceback (most recent call last): File "run_img2txt_dist.py", line 621, in <module> main() File "run_img2txt_dist.py", line 546, in main iter_bar.set_description('Iter (loss=%5.3f)' % loss.item()) ValueError: only one element tensors can be converted to Python scalars

What is the recommended way to run a parallel training.
Thanks :)

Hello, Thanks for your work.

Currently, I am trying to run inference on flickr features. I have installed apex as per the instructions in setup.sh. I have the same commit of apex (1603407bf49c7fc3da74fceb6a6c7b47fece2ef8) as mentioned in the setup.sh file. My pytorch version is 1.6.0+cu101, which is different from the one mentioned in misc/vlp.yml file. While installing apex I get following error message: error: command 'gcc' failed with exit status 1. When I try running the flickr inference code I get below error:

Traceback (most recent call last):
  File "vlp/decode_img2txt.py", line 19, in <module>
    from pytorch_pretrained_bert.tokenization import BertTokenizer, WhitespaceTokenizer
  File "/home/default/ephemeral_drive/work/image_captioning/VLP/pytorch_pretrained_bert/__init__.py", line 6, in <module>
    from .optimization_fp16 import FP16_Optimizer_State
  File "/home/default/ephemeral_drive/work/image_captioning/VLP/pytorch_pretrained_bert/optimization_fp16.py", line 4, in <module>
    from apex.optimizers import FP16_Optimizer
ModuleNotFoundError: No module named 'apex.optimizers'

I tried installing the latest apex version as per the instructions here. I get Successfully installed apex-0.1 message, but when I run the inference code I get below error.

Traceback (most recent call last):
  File "vlp/decode_img2txt.py", line 19, in <module>
    from pytorch_pretrained_bert.tokenization import BertTokenizer, WhitespaceTokenizer
  File "/home/default/ephemeral_drive/work/image_captioning/VLP/pytorch_pretrained_bert/__init__.py", line 6, in <module>
    from .optimization_fp16 import FP16_Optimizer_State
  File "/home/default/ephemeral_drive/work/image_captioning/VLP/pytorch_pretrained_bert/optimization_fp16.py", line 4, in <module>
    from apex.optimizers import FP16_Optimizer
ImportError: cannot import name 'FP16_Optimizer'

Seems, the optimizers are different in the latest apex commit. Would you recommend replacing the optimizer in your code with any of the current ones. Or, if you would have any other suggestion for me to resolve the issue. I am not able to use the conda environment you had mentioned as a requirement, as I am working on a controlled-access machine and I don't have the liberty to do all the installations.

Hello, I am using the COCO dataset, A two-layer LSTM model, one layer for top-down attention, and one layer for language models.

Extracting words with jieba I used all the words in the picture description that occurred more than 3 times as a dictionary file, and a total of 14,226 words. words = [w for w in word_freq.keys () if word_freq [w]> 3]

After training the model, when using it, multiple words of the same type appear in the result, such as:

Note notebook laptop computer on bed A little girl little girl girl standing together

How can I solve this problem?

UniLM checkpoint

wget -O bert_save.tar.gz "https://onedrive.live.com/download?cid=E5364FD183A1F5BB&resid=E5364FD183A1F5BB%212016&authkey=AB5-lxzCkgpfLhg"

The above link is no longer anymore. Thanks.

Thanks a lot for sharing this useful repo. we are trying to reproduce the finetuning result on flicker30k, but an error occurs which say "feat_cls_1000_float16/cc_detection_vg_100dets_gvd_checkpoint_trainval not found". I found this error is about this line of code, parser.add_argument('--region_det_file_prefix', default='feat_cls_1000/coco_detection_vg_100dets_gvd_checkpoint_trainval', type=str) . So where can I get access of "coco_detection_vg_100dets_gvd_checkpoint_trainval"? In my folder of feat_cls_1000, there are only "flickr30k_detection_vg_thresh0.2_feat_gvd_checkpoint_trainvaltest.h5" and "trainval".

I want to change the number of objects in the feature extractor (e.g from 100 to 150) and the attention head (instead of changing the BERT base to BERT large) (e.g from 12 to 16). Could you please tell me where to change the code?

We intend to use our own dataset to pre-train your model, but we don't know how to construct our data to fit into your model. In addition, COCO dataset is too large and we don't have enough space, so we cannot see the JSON file in which you organize your training data. So could you please share your JSON file so that we can know how to use our own data? Thanks a lot!

Hi @LuoweiZhou ， is it convenient to provide other pre-trained checkpoints, such as cc_g8_lr1e-4_batch512_s0.25_b0.75.tar.gz or cc_g8_lr1e-4_batch512_s0_b1.tar.gz ? Many thanks.

Hi, thanks for sharing this project.

I prepared all the required h5py files and caption annotations files for COCO Caption finetuning as instructed in README. The training went normally at the beginning, but got killed (bus error (core dumped)) after around 70k~100k iteration.

I wonder if it was an out-of-memory issue caused by data loading. It seemed that huge memory was progressively consumed by the program, perhaps due to reading more and more image features from h5py files. Using del or gc.collect() didn't help free unreferenced objects' memory.

Is there any good solution to save memory for the multimodal training? Or idea on what was going on in my case. Thanks a lot!

Hi, Thanks for your great work! I want to know whether the model can be trained without regions. In other words, I have caption and image only without any bbox info, how can I make the model work? Thank you so much!