Sir, thank you very much for publicing the code.

Recently I have download all items with the first color from FACAD. The json format of dataset is named "meta_all_129927.json".

When I check the download data, I get 126753 items which are the first color data in FACAD. There are about 1200 images which are lack of link (basically, one product lacks one link) and 3 items do not have any links. Data processing is very troublesome

I use the data of 100000 images, the result is too low and data process must have some problems

Results of the val(use the train data to evaluate): B4: 0.09 M: 0.07 R: 0.11 C: 0.23

So I download the data named 'TEST_IMAGES_5.hdf5,VAL_IMAGES_5.hdf5,TRAIN_IMAGES_5.hdf5' you provide.

For example, the num of images in TEST_IMAGES_5.HDF5 and TEST_IMAGEPATH_5.json are 99981, however, the other file with json format like 'TEST_CAPLENS_5.json', 'TEST_CAPTIONS_5.json' ... are 99946. I only check the test data.

May you check the data upload? I use the TEST_IMAGEPATH_5.json to find image id and get the description, attribute.... I do not know this way is right.

Today, I download the code, find some files is lack in yml such as sat.yml: data_folder: /home/xuewyang/Xuewen/Research/data/FACAD/jsons model_folder: /home/xuewyang/Xuewen/Research/model/fashion/captioning/SAT2 checkpoint: /home/xuewyang/Xuewen/Research/model/fashion/captioning/SAT/vanilla/BEST_checkpoint_2.pth.tar

I sent a few emails, and I got no reply. I am very, very interested in fashion caption and hope to get your help

Thanks for your sharing your dataset. It seems to be a really useful and fantastic work! But I'm getting into troubles when I try to replicate some results.

I used the code in ruotian's repo to try some baselines. I trained the 'att2in' and 'adaatt' model using XE loss on FACAD, but got really bad performance on BLEUs, METEOR, ROUGEL, and CIDEr. Even when I use the training split to evaluate the trained model, the socres are still much lower than reported in the paper except CIDEr.

And I also find that the training loss can drop to 1.8 after epochs of training, while the loss on the val split stops at about 3.1. It seems that I've come across an overfit but I've no idea about the reason, as I think the amout of the data is big enough to avoid overfitting. Note that these models all behave well on COCO dataset. And I think I've preprocessed FACAD into the COCO format.

The only difference is that, in COCO, each image is paired with 5 captions. While in FACAD, each image is paired with only one caption, and sometimes different images share one same caption. I don't know if this difference results in the terrible performance.

Do you have any ideas on these problems? Are there any significant details for data preprocessing or training?

Thanks for your released code. The code is well-structured, I replaced the dataloader with my own implemention and it still works well. But I still got some issues.

I've trained the SAT and BUTD model for about 15 epochs now, they all achieve high scores but the differences are quite large, especially on CIDEr. The CIDEr of them are about 192.6 and 144.6 respectively. Are these results alright? So what scores have you got with these models?

Detailed results are as follows: SAT: Bleu_1: 0.495 Bleu_2: 0.348 Bleu_3: 0.267 Bleu_4: 0.219 METEOR: 0.215 ROUGE_L: 0.465 CIDEr: 1.928

BUTD: Bleu_1: 0.462 Bleu_2: 0.302 Bleu_3: 0.213 Bleu_4: 0.161 METEOR: 0.193 ROUGE_L: 0.432 CIDEr: 1.446

Besides, I've also trained CNNC for 4 epochs. I find it quite slow to eval and it achieves really low scores. Bleu_1: 0.158 Bleu_2: 0.057 Bleu_3: 0.020 Bleu_4: 0.009 METEOR: 0.060 ROUGE_L: 0.131 CIDEr: 0.094

I got some puzzles while reading your paper.

1.How to get the attribute vector z? More specifically, how to transfer the image features into vector z? In the paper, z is obtained from a feed-forward layer, what functions in pytorch did you use to combine this layer? Linear functions or convolutional layers? There could be several strategies to compress the 3-dimension image features into a vector.

2.In formulation 8, the subscript 1/n is put out of the brackets, is it a typo error? Does it mean β * P(1) * √(P(2)) or β * √(P(1) * P(2))？

Is dataset used in the paper different from the preprocessed dataset provided on google drive? Or Am I missing something?
Preprocessed data from the google drive: TRAIN: 888293 VAL: 19915 TEST: 101225

From paper Section 5.1: It contains 993K images and 130K descriptions, and we split the whole dataset, with approximately 794K image-description pairs for training, 99K for validation, and the remaining 100K for test.

Hi, very cool work, thanks a lot for making your code public! it would be great if you could share a pre-trained model + sample script to create captions on new images. Would be super cool and helpful.

Would that be possible? Thanks! Z.

Thank you for your works. It looks interested and useful works. but I found some problems in the step of using this new dataset and training step. I need you to explain what I should do such as:

1. What files should I download for training the model?
2. Which library I should install?
3. etc.

Could you explain it for me, please Looking forward to your reply @xuewyang, Thank you.

Thank you. I meet some problems when duplicating the model.

1. Like the " the encoder is a pre-trained CNN, which takes an image as the input and extracts B image features, X={x0, x1,...xB}. Is that X means the feature map (batchsize* 2048* 14* 14) outputs from the last convolution layer of resnet101?

2.Like the figure3, the avg pooling of the feature map (batchsize * 2048) is sent into the feed forward networks. How many layers consist in the FF, only one layer (2048 * 990) followed by a sigmoid or more layers? Is the output of the FF before sigmoid chosen as the Z or after sigmoid chosen as the Z?

3. Whether the attribute learning has been pretrained seperately before and then added to fintune the caption model or attribute learning and caption model are learning together from the beginning?

4.When we get z, whether z is cat with y(word embedding) as input of caption model or z cat with the output of image features after attention model?

Looking forward to your reply.