Hi, I followed the data preparation and ran the training script for the default CLIP-RN50 model in the readme. However, the training job crashes with the log below. Could you please check if the current example training script is runnable?

$ /scratch-space/CLIP-ViL/CLIP-ViL-Direct/caption 
> python tools/train.py --cfg configs/phrase1/clip_rn50_transformer_scl.yml
Warning: key N_enc not in args
Warning: key N_dec not in args
Warning: key d_model not in args
Warning: key d_ff not in args
Warning: key num_att_heads not in args
Warning: key dropout not in args
Warning: key REFORWARD not in args
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocotalk_clip_RN50_fc data/cocotalk_clip_RN50_att data/cocotalk_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
Read data: 0.0007147789001464844
/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:65: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
iter 0 (epoch 0), train_loss = 9.158, time/batch = 25.627
Read data: 0.0002460479736328125
iter 1000 (epoch 0), train_loss = 4.920, time/batch = 0.183
Read data: 0.00023293495178222656
iter 2000 (epoch 0), train_loss = 3.784, time/batch = 0.194
Traceback (most recent call last):
  File "tools/train.py", line 293, in <module>
    train(opt)
  File "tools/train.py", line 246, in train
    val_loss, predictions, lang_stats = eval_utils.eval_split(
  File "/scratch-space/CLIP-ViL/CLIP-ViL-Direct/caption/captioning/utils/eval_utils.py", line 171, in eval_split
    seq, seq_logprobs = model(fc_feats, att_feats, att_masks, opt=tmp_eval_kwargs, mode='sample')
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 167, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 177, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in replica 5 on device 5.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/scratch-space/CLIP-ViL/CLIP-ViL-Direct/caption/captioning/models/CaptionModel.py", line 33, in forward
    return getattr(self, '_'+mode)(*args, **kwargs)
TypeError: _sample() missing 2 required positional arguments: 'fc_feats' and 'att_feats'

In CLIP-ViL/CLIP-ViL-Direct/caption/clip.py, "x = x + self.positional_embedding[0, :, None, :].to(x.dtype) # (HW+1)NC" should be " x = x + self.positional_embedding[:, None, :].to(x.dtype) # (HW+1)NC"

Could you explain why you change the following code of the official clip? In official clip, out_proj_weight=self.c_proj.weight, out_proj_bias=self.c_proj.bias. In your code,

Hi! @airsplay. I have used precomute_imagenet_views.py and found that the result is not consistent with the tsv file provided by https://nlp.cs.unc.edu/data/vln_clip/features/CLIP-ViT-B-32-views.tsv . I wonder that how did you get the tsv file provided on that website? I run it with the default settings(arch=vit, LABEL=False)

Thanks for your open source code! I want to inquire about the training time of Pythia utilizing your CLIP features.

It seems to take me more than a week if I use mmf with your config for Pythia (even setting the batchsize to 32 with original iterations). However, when I utilize the code in https://github.com/KaihuaTang/VQA2.0-Recent-Approachs-2018.pytorch and run the butp, I could finish the training in 5-6 hours with 4 Nvidia 2080Ti.

To be honest, I am not familiar with mmf. And I just want to make sure whether I did something wrong.

Hi, @airsplay , @sIncerass , @liunian-harold-li , @clip-vil

Thanks for sharing your code! This is a very interesting work :)

However, I find that there may be an inappropriate operation in ./CLIP-ViL-Direct/caption/scripts/clip_prepro_feats.py when extracting the visual CLIP features. Specifically, when you resize the official CLIP models' positional_embedding to support larger image input resolution than the official CLIP models, you assign pos_embed.weight with the resized pos_embed weights while the pos_embed.data actually used in forward() remains zeros. I guess the lack of positional_embedding's weights when extracting the CLIP grid features is the true reason for the large performance degradation on the experiments using CLIP-ViT-B grid features. The performances on CLIP ResNet models are relatively normal because the model.visual.attnpool.positional_embedding layer is only used on the top of CLIP ResNet models to aggregate the extracted grid features for the global visual feature.

To verify my guess, I re-implement a new feature-extraction pipeline and run the COCO Captioning task using X-modaler codebase with the extracted CLIP grid features. I get 127.8 CIDEr score using ViT-B/32_448 CLIP grid features. I have provided my detailed experiment results on COCO Captioning here.

So I wonder whether the lack of positional_embedding's weights when extracting CLIP-ViT-B grid features also destroyed the performance on VQA tasks? And the visualization and conclusion in Figure3 will change after providing the pos_embed_weights when extracting CLIP-ViT-B grid features?

Best, Jianjie

Hi! I'm trying to run mcan feature extraction.

This is the command I was trying to run: python mcan_clip_grid_feature.py --config-file configs/R-50-grid.yaml --dataset coco_2014_train --model_type RN50. And I got the following error. I searched throughout the codebase and didn't find where self.tfm_gen is defined.

Traceback (most recent call last):
  File "mcan_clip_grid_feature.py", line 142, in <module>
    main(args)
  File "mcan_clip_grid_feature.py", line 136, in main
    do_feature_extraction(cfg, model, args.dataset, args)
  File "mcan_clip_grid_feature.py", line 78, in do_feature_extraction
    extract_clip_feature_on_dataset(model, data_loader, dump_folder, args)
  File "mcan_clip_grid_feature.py", line 113, in extract_clip_feature_on_dataset
    for idx, inputs in enumerate(tqdm.tqdm(data_loader)):
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/detectron2/data/common.py", line 90, in __getitem__
    data = self._map_func(self._dataset[cur_idx])
  File "/nlp/data/weiqiuy/miniconda3/envs/mmf/lib/python3.7/site-packages/detectron2/utils/serialize.py", line 26, in __call__
    return self._obj(*args, **kwargs)
  File "/mnt/nlpgridio3/data/weiqiuy/CLIP-ViL/CLIP-ViL-Direct/vqa/grid_feats/dataset_mapper.py", line 107, in __call__
    self.tfm_gens = [self.tfm_gens[0]] + [T.Resize((600, 1000))]
AttributeError: 'AttributeDatasetMapper' object has no attribute 'tfm_gens'

I'm trying to extract image features for vqa with pythia using python pythia_clip_grid_feature.py --config-file configs/R-50-grid.yaml --dataset coco_2015_train --model_type RN50. Isn't this supposed to give output as 100 object features of 2048 dimension. I'm getting varying outputs of dimensions, (1, 13, 20, 2048), (1, 15, 20, 2048) and similar ones. Could anyone point out where I'm wrong and where I need to make changes to get (100, 2048) output for an image. And what should be the format of annotation file if I am to use VQA dataset, since that doesn't have attributes like area, segmentation, categories, etc Thanks

Hello, thanks for sharing such a great code base. If I want to extract VQAv2's COCO image features (or some other dataset) by using mcan_clip_grid_feature.py, how should I prepare the data format and specify the dataset directory that the code needs ?

Thank you for your answer to the previous question. The following errors occurred when I used resnet model to extract clip features. The dimension of x is 3, the dimension of self.positional_embedding is 2, but the dimension of slice operation is 4. I don't know what was wrong and I hope to get your help. Thank you very much.

Thanks for sharing your code! When I run clip_prepro_feats.py, the following error occurred. I tried many times, but still failed. I hope to get your help.

Hi @sIncerass , thanks for your interesting work. I have a question about image captioning results in Table 2. I find that the Transformer model with CLIP-ViT-B feature can still get a good performance instead of the dramatically worse performance reported in Table 2. Maybe there is a bug in the CLIP-ViT-B feature extraction.

Thanks for your open source code! I'm trying the reproduce the results of experiments in Table 7 in the ICLR paper: 'Zero-Shot Performance of CLIP in VQA'.

Would you provide code or detail information for this?

It seems that the model uses different annotation files from the code. What's the difference between it different and the original SNLI-VE jsonl file？And where can I found it? Can you share it with us? Thank you in advance!

# CLIP-ViL-Pretrain/src/tasks/snli_data.py
text_db_paths = {
    "valid": "/local/harold/ubert/clip_vlp/lxmert/data/snli_ve/txt_db/ve_dev.db",
    "train": "/local/harold/ubert/clip_vlp/lxmert/data/snli_ve/txt_db/ve_train.db",
    "test": "/local/harold/ubert/clip_vlp/lxmert/data/snli_ve/txt_db/ve_test.db",
}

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Hi, just wanted to ask if there is a checkpoint for the finetuned model on SNLI-VE and if it would be possible share it? Thank you in advance! @airsplay