I use the original flickr30k dataset rather than precomputed features ,and use the pretrained model resnet50.I set finetune True. But, i get poor result of recall. R@1,R@5,R@10 is nearly zero.I can't find the reason about the bad result.How do you set the parameters when you use resnet with finetune? Thank you.

Hi,

First of al, thanks for sharing this great work!

I've difficulties reproducing the results from the paper as a baseline. I will talk about experiment #3.15 in this issue: VSE++ (ResNet), Flickr30k

So what I get from the paper, the config is the following:

• 30 epochs
• Load images from disk, no precomputed features?
• lower the lr after 15 epochs.
• lr goes from 0.0002 -> 0.00002

My question is: is the image-encoder here trained end-to-end or not. In other words, is ResNet152 only used as a fixed feature extractor, or is it optimized?

According to your documentation, VSE++ (and therefore I assume 3.14) can be reproduced by - only - using the --max_violation flag, but I get (way) lower results, do I need the --finetune flag as well?

Thanks, Maurits

The code in 176, 177 line of train.py:

if model.Eiters % opt.val_step == 0:  
    validate(opt, val_loader, model)

will run the validation and call model.val_start() to stop the batch_norm and dropout layers of img_encoder and txt_encoder.

However, it will not call model.train_start() until next epoch, that may cause the dropout and batch_norm layers will only activate in the first val_step steps for each epoch...

Is it a bug need to be fixed?

This is a PR as much as it is a question, according to the documentation on pack_padded_sequence, by default it understands that their input comes sorted from longest sentence to shortest.

        enforce_sorted (bool, optional): if ``True``, the input is expected to
            contain sequences sorted by length in a decreasing order. If
            ``False``, the input will get sorted unconditionally. Default: ``True``.

I cannot see where is this guaranteed at data loading time to ensure that every batch is properly processed, could this be an issue? I have reproduced your results, but I have seen differences when embedding sentences alone or in a batch, and what I have observed is that embeddings in a batch seemed to be mixed up.

I want to train "coco" instead of "coco_precomp", but the file indicated by "download.sh":

http://msvocds.blob.core.windows.net/annotations-1-0-3/captions_train-val2014.zip

doesn't contain the coco_train/val/test_ids.npy files and I encountered No such file or directory: vsepp/data/coco/annotations/coco_train_ids.npy

So where to get these files in order to train "coco"?

Thanks in advance.

This code works quite well. Thanks for sharing it. I'm wondering, do you have any code snippets to show how one might use a trained VSE++ model to create their own caption query from text (i.e. a string), submit it to the VSE++ model to get a single caption embedding and then search for matching images that have also mapped to the joint space using the same model? It's easy to do the comparison once a numpy array for the caption and image embeddings in joint space are created, but it's not clear how to use your model with a brand new caption query or simply a set of CNN image features that are not part of some complete COCO/FLICKR/etc train or test set with corresponding caption/image pairs. Thanks for any tips. I'd prefer not to rewrite everything if you already have some additional tools for this.

diff --git a/data.py b/data.py
index 913ea16..520c38d 100644
--- a/data.py
+++ b/data.py
@@ -221,13 +221,12 @@ class PrecompDataset(data.Dataset):
     def __getitem__(self, index):
         # handle the image redundancy
         img_id = index/self.im_div
-        image = torch.Tensor(self.images[img_id])
+        image = torch.Tensor(self.images[int(img_id)])
         caption = self.captions[index]
         vocab = self.vocab
 
         # Convert caption (string) to word ids.
-        tokens = nltk.tokenize.word_tokenize(
-            str(caption).lower().decode('utf-8'))
+        tokens = nltk.tokenize.word_tokenize(str(caption).lower())
         caption = []
         caption.append(vocab('<start>'))
         caption.extend([vocab(token) for token in tokens])
diff --git a/evaluation.py b/evaluation.py
index 7e5da4e..9171f85 100644
--- a/evaluation.py
+++ b/evaluation.py
@@ -57,7 +57,7 @@ class LogCollector(object):
         """Concatenate the meters in one log line
         """
         s = ''
-        for i, (k, v) in enumerate(self.meters.iteritems()):
+        for i, (k, v) in enumerate(self.meters.items()):
             if i > 0:
                 s += '  '
             s += k + ' ' + str(v)
@@ -66,7 +66,7 @@ class LogCollector(object):
     def tb_log(self, tb_logger, prefix='', step=None):
         """Log using tensorboard
         """
-        for k, v in self.meters.iteritems():
+        for k, v in self.meters.items():
             tb_logger.log_value(prefix + k, v.val, step=step)
 
 

def t2i(images, captions, npts=None, measure='cosine', return_ranks=False): """ Text->Images (Image Search) Images: (5N, K) matrix of images Captions: (5N, K) matrix of captions """ if npts is None: npts = int(images.shape[0] / 5) ims = numpy.array([images[i] for i in range(0, len(images), 5)])

why divide 5？

Hi. Thanks for your code. 1- May I ask why are you including the start and end token when constructing the caption? Since you want to encode the caption only, there is no need for it. As far as I know, the start and end token are only needed when predicting text (such as image captioning, neural machine transaltion....etc). But for your case, you just want to encode. Or does it have to do with how the evaluation metric is calculated?

2- I also have a question about the data loader. In this part:

        if self.images.shape[0] != self.length:
            self.im_div = 5
        else:
            self.im_div = 1
        # the development set for coco is large and so validation would be slow
        if data_split == 'dev':
            self.length = 5000

I understand that for the training and test splits, you are replicating the image 5 times (the number of captions per image). However, for the 'dev' split (validation after training), you are specifying 5000 only. For Flickr30k, that would still be correct (since we have 1000 validation images * 5). But for COCO, the actual validation dataset with the replication is 25K. But you are loading only a portion of it. According to how the data loader works, it will generate indices according to the length of the dataset specified in __len__. Therefore, for COCO dev set, it will generate 5000 indices, and with images[i//5], this is will retrieve only 1000 original COCO validation images. So my question, is that right to be done? What if the other samples are better? This would lead to a low validation score while it should be high.

I have read some image-text matching papers but I still have no idea about how to select 1k images from the 5k test images, how can I get the ID list of the 1k test images?

Hi @fartashf,

Great code base, very easy to work with! I had two quick questions regarding evaluation metrics:

• I noticed that meanr increases as rsum increases and was wondering if you had an explanation for this? (See plots below) I should mention that these results are with a few modifications to your code. Specifically, I used Glove embeddings where the embeddings are not trained / "backproped" into.

• Also was wondering what was the reason for choosing rsum for model selection instead of meanr?

Thanks!

I have a question about your model. I have a image with pill object and list of text is name of pill corresponding to image. I want to map pill object and name of it. Because number pill classes very huge so I know some prior infomation about classes . I have train faster RCNN to detect pill and now i must map it with name. Is your model is relavent to this problem. Thank you so much