Thanks for opening source code. The provided pretrained model can perform the results claimed in your paper, but when I fail to training model on flicker dataset to reproduce the results.

Hi, I am trying to reproduce the baseline(Topdown in Flickr 30k, with CIDEr optimization in RL phase) in your paper, and I used different kinds of code including yours, but I failed, The result of CIDEr score is around 60, can not reach the similar result in your paper (over 67), can you share the detailed training setting?

Thanks!

您好，我是北京理工大学计算机学院的一名学生，我最近在学习您2020年发布在cvpr的论文Moer Grounded Image Captioning by Distilling Image-Text Matching Model，在我复现论文代码的时候我按照readme的要求配置好了环境，并顺利进行了代码的评估部分，但是在我训练模型的过程中缺少了/log/CE-gt-sup-0.1-nll/infos_CE-gt-sup-0.1-nll.pkl文件，而/log/CE-gt-sup-0.1-nll文件夹下只有infos_CE-gt-sup-0.1-nll-best.pkl文件，报错信息如下：

然后我以为两个文件是相同的，因此将infos_CE-gt-sup-0.1-nll-best.pkl，model-best.pth更改为代码中的infos_CE-gt-sup-0.1-nll.pkl，model.pth,但是又报以下错误 因此我想请教您log.tar.gz是否缺少了这两个infos_CE-gt-sup-0.1-nll.pkl，model.pth,这两个文件。期待您的指导，非常感谢您。

Hi, thanks for your work. I'm curious to use your code on new raw images, how can I do that? From the vis_attn_cap.ipynb, I modified this part : opt = parser.parse_args(args=["--image_folder","my_images_folder"]) And it use dataloaderraw

The problem is dataloaderraw doesn't yield box_feats. How do I get the box_feats? Thanks :)

Hi,

I run your evaluation results on both F50k and Coco. The results of F30k is absolutely right. But the results on coco is not quite ok. The only detail I find strange is misc/SCAN/runs/f30k_SCAN_POS1. Could you please offer me the model of misc/SCAN/runs/coco... or something else I made some error? Many thanks.

Results Reported in Table 4. Eval Up-Down+XE Constructing SCAN model... scan_model_path:misc/SCAN/runs/f30k_SCAN_POS1/checkpoint/model_best.pth.tar Done DataLoader loading json file: data/cocotalk.json vocab size is 9487 DataLoader loading h5 file: data/mscoco/cocobu_fc data/mscoco/cocobu_att data/mscoco/cocobu_box data/cocotalk_label.h5 max sequence length in data is 16 read 123287 image features assigned 113287 images to split train assigned 5000 images to split val assigned 5000 images to split test loading annotations into memory... Done (t=0.59s) creating index... index created! using 5000/5000 predictions Loading and preparing results... DONE (t=0.05s) creating index... index created! tokenization... PTBTokenizer tokenized 307085 tokens at 801422.82 tokens per second. PTBTokenizer tokenized 51973 tokens at 244262.23 tokens per second. setting up scorers... computing Bleu score... {'testlen': 46974, 'reflen': 46862, 'guess': [46974, 41974, 36974, 31974], 'correct': [35147, 19145, 9763, 4961]} ratio: 1.0023899961589133 Bleu_1: 0.748 Bleu_2: 0.584 Bleu_3: 0.448 Bleu_4: 0.344 computing METEOR score... METEOR: 0.269 computing Rouge score... ROUGE_L: 0.554 computing CIDEr score... CIDEr: 1.084

Hi,

I am trying to use your codebase for some experiments and during training I get the following error:

$ python train.py --id CE-scan-sup-0.1kl --caption_model topdown --input_json data/flickrtalk.json --input_fc_dir data/flickrbu/flickrbu_fc --input_att_dir data/flickrbu/flickrbu_att --input_box_dir data/flickrbu/flickrbu_box --input_label_h5 data/flickrtalk_label.h5 --batch_size 29 --learning_rate 5e-4 --learning_rate_decay_start 0 --scheduled_sampling_start 0 --checkpoint_path log/CE-scan-sup-0.1kl --save_checkpoint_every 1000 --val_images_use -1 --max_epochs 30 --att_supervise True --att_supervise_weight 0.1 Constructing SCAN model... scan_model_path:misc/SCAN/runs/f30k_SCAN_POS1/checkpoint/model_best.pth.tar Done tensorboardX is not installed DataLoader loading json file: data/flickrtalk.json vocab size is 7000 DataLoader loading h5 file: data/flickrbu/flickrbu_fc data/flickrbu/flickrbu_att data/flickrbu/flickrbu_box data/flickrtalk_label.h5 max sequence length in data is 16 read 31014 image features assigned 29000 images to split train assigned 1014 images to split val assigned 1000 images to split test Read data: 0.7768440246582031 Traceback (most recent call last): File "train.py", line 291, in <module> train(opt) File "train.py", line 180, in train model_out = dp_lw_model(fc_feats, att_feats, labels, masks, att_masks, data['gts'], torch.arange(0, len(data['gts'])), sc_flag, box_inds) File "/users/vlad/anaconda3/envs/gvd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/users/vlad/anaconda3/envs/gvd/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/users/vlad/anaconda3/envs/gvd/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/users/vlad/anaconda3/envs/gvd/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply raise output File "/users/vlad/anaconda3/envs/gvd/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker output = module(*input, **kwargs) File "/users/vlad/anaconda3/envs/gvd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/users/vlad/image_grounding/Grounded-Image-Captioning/misc/loss_wrapper.py", line 45, in forward _, grd_weights,noun_mask= get_self_critical_reward(self.model, fc_feats, att_feats, att_masks, gts, labels[:,1:].detach(), vars(self.opt)) File "/users/vlad/image_grounding/Grounded-Image-Captioning/misc/rewards.py", line 68, in get_self_critical_reward gts = {i: gts[i % batch_size // seq_per_img] for i in range(2 * batch_size)} File "/users/vlad/image_grounding/Grounded-Image-Captioning/misc/rewards.py", line 68, in <dictcomp> gts = {i: gts[i % batch_size // seq_per_img] for i in range(2 * batch_size)} KeyError: 29 Terminating BlobFetcher

I followed all the steps described and I am able to run the evaluation, but for training I get the above error. I did some investigations and I think a reshape is missing somewhere since the dataloader loads multiple captions per image but this doesn't seem to be reflected here https://github.com/YuanEZhou/Grounded-Image-Captioning/blob/77295a6e36de817f173435e809effc3396469ee3/misc/rewards.py#L70 Any suggestions on how to overcome this?

Thanks!