Hi,

Thank you for releasing the code! It's an interesting project!

I'd like to know is it available to also release the pretraining logs? Or some milestones to verify the training process?

Thanks! Blakey

Thanks for the great work! I just want to confirm in this project, the released pre-trained models are also with https://github.com/zengyan-97/X-VLM/blob/master/LICENSE Thanks a lot!

Is it possibile to run itr_flickr as not distributed but on a single gpu?

When running: python run.py --task "itr_flickr" --dist "gpu0" --output_dir "output/itr_flickr" --checkpoint "4m_base_finetune/itr_flickr/checkpoint_best.pth"

I get:

Training Retrieval Flickr

| distributed init (rank 0): env:// Traceback (most recent call last): File "Retrieval.py", line 381, in main(args, config) File "Retrieval.py", line 215, in main utils.init_distributed_mode(args) File "C:\Users..\X-VLM-master\utils_init_.py", line 357, in init_distributed_mode world_size=args.world_size, rank=args.rank) File "C:\Users..\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\distributed\distributed_c10d.py", line 434, in init_process_group init_method, rank, world_size, timeout=timeout File "C:\Users..\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\distributed\rendezvous.py", line 82, in rendezvous raise RuntimeError("No rendezvous handler for {}://".format(result.scheme)) RuntimeError: No rendezvous handler for env://

Hi! @zengyan-97

I can not find the check point that can be finetuned.

Thank you for providing these check points:

• nlvr_domain_pretrain.th
• 16m_base_model_state_step_199999.th
• 4m_base_model_state_step_199999.th

Also there is a ckpt that you have already finetuned:

• nlvr_ft/checkpoint_best.pth

However, can you guide me for how to finetune on NLVR2?

Hello,

I wonder how did you finetune your model for the Flickr30K dataset? Did you freeze The Text and Vision Encoder on only fine-tuned the itm_head, or did you apply the fine-tuning to the whole model?

Dear authors: Thanks for your great works. While pretraining the model, I don't know how to organize the open dateset. I have already tried the following yaml.

I am confused about the "lm_domain_pretrain.th" file in "4m_base_finetune/coco_caption/". If I want to reproduce the fine-tuning results on coco captioning, which pre-trained model should I load: "lm_domain_pretrain.th" or "4m_base_model_state_step_199999.th"? Maybe you could provide the specific commands of two-stage finetuning on coco captioning? Thanks!

Hi,

Thanks for the great work.

In Figure 3 of your paper, you showed the Grad-CAM visualizations of your model on RefCOCO+ from text descriptions. Could you share the code for using Grad-CAM on your model?

Thanks!

Thanks for your code.

I note that in your paper, you said "We sample the data by making half of the images in a batch containing bounding box annotations".

But the code is: https://github.com/zengyan-97/X-VLM/blob/e7b960256d194952321b5adad39770c03e6ce9c2/Pretrain.py#L82-L121

The iter_perc you used is 0.5, which means only 50% of time, the model takes a batch of image-text-box data and a batch of image-text data as input; and otherwise, the model only takes a batch of image-text data as input.

Therefore, it seems that iter_perc = 1.0 fits your statement in the paper.

According to the ablation study results in Table 4, you have certainly tested the impact of iter_perc = 0.0 (corresponding to the model, X-VLM w/o all).

So, have you tested more values of iter_perc (e.g., 1.0)?

Thanks for your great sharing.

https://github.com/zengyan-97/X-VLM/blob/e7b960256d194952321b5adad39770c03e6ce9c2/models/xvlm.py#L121

As shown above, you mentioned in the code that initilaizing the vision encoder with deit is worser than clip-vit and swin.

Do you have some supporting results? For example, the performance on Image-Text Retrieval with deit or swin

Traceback (most recent call last): File "VQA.py", line 283, in main(args, config) File "VQA.py", line 134, in main train_dataset, vqa_test_dataset = create_dataset('vqa', config, args.evaluate) File "/home/deer/X-VLM/dataset/init.py", line 84, in create_dataset vqa_test_dataset = vqa_dataset(config['test_file'], test_transform, config['vqa_root'], config['vg_root'], File "/home/deer/X-VLM/dataset/vqa_dataset.py", line 31, in init tokenizer = RobertaTokenizer.from_pretrained(text_encoder) if use_roberta else BertTokenizer.from_pretrained(text_encoder) File "/home/deer/anaconda3/envs/xvlm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1651, in from_pretrained fast_tokenizer_file = get_fast_tokenizer_file( File "/home/deer/anaconda3/envs/xvlm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3467, in get_fast_tokenizer_file all_files = get_list_of_files( File "/home/deer/anaconda3/envs/xvlm/lib/python3.8/site-packages/transformers/file_utils.py", line 1818, in get_list_of_files model_info = HfApi(endpoint=HUGGINGFACE_CO_RESOLVE_ENDPOINT).model_info( File "/home/deer/anaconda3/envs/xvlm/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 94, in _inner_fn return fn(*args, **kwargs) File "/home/deer/anaconda3/envs/xvlm/lib/python3.8/site-packages/huggingface_hub/utils/_deprecation.py", line 98, in inner_f return f(*args, **kwargs) File "/home/deer/anaconda3/envs/xvlm/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1289, in model_info hf_raise_for_status(r) File "/home/deer/anaconda3/envs/xvlm/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 242, in hf_raise_for_status raise RepositoryNotFoundError(message, response) from e huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Tpjmanp6E_vsjeLwXkyXg)

Repository Not Found for url: https://huggingface.co/api/models/data/bert-base-uncased. Please make sure you specified the correct repo_id and repo_type. If the repo is private, make sure you are authenticated.

Hi, have you ever had this problem? how to deal with this problem.

Currently I want to train a model with 480 resolution, but I cannot find the link to download the image encoder swin-480. Can you share me the link to download it? Thank you very much.