I was following the instructions on this page regarding how to use the builtin mscoco dataset for image captioning. I have doubts on the following points:

1. In which directory should i have my images and annotations?
2. I was trying to run the tools/create_feats.py script for converting the karpathy_train_resnet101_faster_rcnn_genome.tsv.0 into .npz format. however I am running out of space on colab pro. What could be the reason for this and is there any fix available?

when i run up-down config，the processing is well but when i run xlan config，there are some errors in the procssing of training. I change num_workers=0, this error still exsit. thanks for your reply Error################################### eta: 3:10:12 iter: 800 total_loss: 2.673 time: 1.4086 data_time: 0.4650 lr: 2.5031e-05 max_mem: 2680M Traceback (most recent call last): File "train_net.py", line 71, in args=(args,), File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/launch.py", line 83, in launch daemon=False, File "/data1/wlx/anaconda3/envs/xmodaler/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/data1/wlx/anaconda3/envs/xmodaler/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes while not context.join(): File "/data1/wlx/anaconda3/envs/xmodaler/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 150, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/data1/wlx/anaconda3/envs/xmodaler/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args) File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/launch.py", line 129, in _distributed_worker main_func(*args) File "/data1/wlx/project2021/xmodaler-master/train_net.py", line 59, in main return trainer.train() File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/defaults.py", line 365, in train super().train(self.start_iter, self.max_iter) File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/train_loop.py", line 152, in train self.after_step() File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/train_loop.py", line 182, in after_step h.after_step() File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/hooks.py", line 407, in after_step self._do_eval(epoch) File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/hooks.py", line 372, in _do_eval results = self._func(epoch) File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/defaults.py", line 324, in test_and_save_results eval_results = self.test(self.cfg, self.model, self.test_data_loader, self.test_evaluator, epoch) File "/data1/wlx/project2021/xmodaler-master/xmodaler/engine/defaults.py", line 478, in test eval_res = evaluator.eval(results, epoch) File "/data1/wlx/project2021/xmodaler-master/xmodaler/evaluation/coco_evaler.py", line 44, in eval cocoEval.evaluate() File "/data1/wlx/project2021/xmodaler-master/cococaption/pycocoevalcap/eval.py", line 38, in evaluate gts = tokenizer.tokenize(gts) File "/data1/wlx/project2021/xmodaler-master/cococaption/pycocoevalcap/tokenizer/ptbtokenizer.py", line 55, in tokenize stdout=subprocess.PIPE) File "/data1/wlx/anaconda3/envs/xmodaler/lib/python3.7/subprocess.py", line 800, in init restore_signals, start_new_session) File "/data1/wlx/anaconda3/envs/xmodaler/lib/python3.7/subprocess.py", line 1482, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory

Thank you for your contribution. I read the datacode about COCO carefully. I have a little confused. As for ext_data, how can I get these data ? Looking forward to your reply. ext_data = { "relation": _load_pkl_file(self.relation_file), "attribute": _load_pkl_file(self.attribute_file), "gv_feat": _load_pkl_file(self.gv_feat_file) }

Hi I am using pretrained lstmA3 model. After loading the model successfully and obtaining the dataloader, i am trying to evaluate the model on MSCOCO. I have followed the steps as mentioned in the documentation. However there seems to be some problem with it. Here is the screenshot of the same - from xmodaler.evaluation.coco_evaler import COCOEvaler evaluator = COCOEvaler(cfg, annfile="/content/captions_val2014.json", output_dir="/content/outputs") print(type(dataloader)) model.eval() with torch.no_grad(): for batch in dataloader: print(batch[0].keys()) print(batch[0]["ATTRIBUTE"].shape) output = model(batch) break

I am able to obtain the dataloader from build_xmodaler_valtest_loader. A batch of this loader contains list of dictionaries.

When training with cosnet_rl.yaml, this error occurs: ...... (predictor): BasePredictor( (logits): Linear(in_features=512, out_features=10200, bias=True) (dropout): Dropout(p=0.5, inplace=False) ) (greedy_decoder): GreedyDecoder() (beam_searcher): BeamSearcher() ) [09/08 16:48:14] xl.datasets.common INFO: Serializing 113287 elements to byte tensors and concatenating them all ... [09/08 16:48:23] xl.datasets.common INFO: Serialized dataset takes 162.52 MiB [09/08 16:48:23] xl.datasets.common INFO: Serializing 5000 elements to byte tensors and concatenating them all ... [09/08 16:48:23] xl.datasets.common INFO: Serialized dataset takes 2.68 MiB [09/08 16:48:23] xl.datasets.common INFO: Serializing 5000 elements to byte tensors and concatenating them all ... [09/08 16:48:23] xl.datasets.common INFO: Serialized dataset takes 2.68 MiB [09/08 16:48:39] fvcore.common.checkpoint INFO: [Checkpointer] Loading from /home/stormai/userfile/caoshan/code/xmodaler/ModelResult/xe/cosnet_xe.pth ... [09/08 16:48:44] xl.engine.train_loop INFO: Starting training from iteration 0 [09/08 16:48:45] xl.engine.train_loop ERROR: Exception during training: Traceback (most recent call last): File "/home/stormai/userfile/caoshan/code/xmodaler-master/xmodaler/engine/train_loop.py", line 151, in train self.run_step() File "/home/stormai/userfile/caoshan/code/xmodaler-master/xmodaler/engine/rl_mean_trainer.py", line 47, in run_step bs_rewards = self.scorer(bs_outputs_dict) File "/home/stormai/userfile/caoshan/code/xmodaler-master/xmodaler/scorer/base_scorer.py", line 66, in call gts = [self.gts[i] for i in ids] File "/home/stormai/userfile/caoshan/code/xmodaler-master/xmodaler/scorer/base_scorer.py", line 66, in gts = [self.gts[i] for i in ids] TypeError: list indices must be integers or slices, not numpy.str_

I try to fix this by forcing ids and gts to convert to list, but it doesn't work. Could you pls help me ?

I'm trying to get captions from a pre-trained model on a new dataset (crisismmd) and was wondering whether I need to extract the image features from the raw images first? It looks as though "kfg" utilizes features and their locations rather than raw images as I saw in "mscoco.py":

Could I use a model like detectron2 to get these features, if that is what is needed? Or what would be the optimal way of doing this? Thank you in advance!

Hello, after I accidentally interrupted the training, I used --resume to resume the training, but I found that the learning rate has not been restored. How can I correct such an error. In addition, which parameters control the end time of warm up? Thanks a lot！

Hi,

Thank you very much for your great code. May I ask if you would provide the extracted and pre-processed features (the .npz files) for datasets such as COCO, VCR, MSVTT?

Thank you very much!

Dear contributors, is there a simple way I can directly predict results on my customized input? For example, in the video captioning, I want to generate the caption for my own video, which is in MP4 format. Is there a simple way to do it? Thanks.

您好！非常感谢您的工作。在使用新数据集在cosnet上训练时，产生了一些问题，向您请教一下： 1.feature中使用的CLIP_RN101_49是怎么得到的？我可以自己使用CLIP的编解码器进行处理么？ 2.cosnet中的pkl文件里，'attr_pred' 'attr_labels' 'missing_labels'分别是什么意思？您是否有计划上传预处理的代码？ 非常感谢！

Hi, dear author, in your latest CVPR2022 paper (Comprehending and Ordering Semantics for Image Captioning), how to retrieve semantically similar sentences for the input image using clip model, can you give some tutorials? Thanks a lot!

I have used the train_net.py, but some problems appeared, first i got the 'line 68, in forward wt.index_copy(0, ind, torch.multinomial(prob_prev, 1).view(-1).index_select(0, ind)) RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)' ,but when i cleared the memory and re-run the train_net_py，it could work successfully until the 10 epoch In 10th epoch,i got this: File "D:\llk\xmodaler-master\xmodaler\modeling\meta_arch\rnn_att_enc_dec.py", line 68, in forward wt.index_copy(0, ind, torch.multinomial(prob_prev, 1).view(-1).index_select(0, ind)) #####bug RuntimeError: probability tensor contains either inf, nan or element < 0 Can somebody help me??😥

Thank you for sharing this great work The folder of MSVD dataset features in google drive is empty for these links: (https://drive.google.com/drive/folders/1vx9n7tAIt8su0y_3tsPJGvMPBMm8JLCZ?usp=sharing) (https://drive.google.com/drive/folders/1-jvt6aKMDmhZC03DPEpwwgYxeL4PSD5J) I need the script file for extracting the MSVD features to extract these features for videos by myself, can you send it to me please ([email protected])

I would need to generate a caption in a video, what would be the easiest way to use it? I don't have to train it, just use it to generate this caption. I haven't found a working example in the documentation. Could it possibly be possible to have a colab notebook ready to use?

首先感谢您们的工作，我在尝试使用MSVD的预训练模型权重在我自己准备的数据集上进行视频摘要推理。

我首先使用resnet152提取我自己的数据集的高维特征，然后组装成npy。 之后按照msvd数据集中的caption_test.json，填入我自己数据集中相应的id和file_name，（因为我只要进行推理任务，而且不需要评估，所以annotation为空）。

之后将正确的路径填写到config/video_caption/msvd/base_caption.yaml，但是现在它运行报错，没有1297.npy这个文件。 所以我想提问，为什么我修改了caption_test.json，它还是不读取我的caption_test.json里的数据？

Results: https://github.com/cclauss/xmodaler/actions

https://flake8.pycqa.org/en/latest/user/error-codes.html

On the flake8 test selection, this PR does not focus on "style violations" (the majority of flake8 error codes that psf/black can autocorrect). Instead, these tests focus on runtime safety and correctness:

• E9 tests are about Python syntax errors usually raised because flake8 can not build an Abstract Syntax Tree (AST). Often these issues are a sign of unused code or code that has not been ported to Python 3. These would be compile-time errors in a compiled language but in a dynamic language like Python, they result in the script halting/crashing on the user.
• F63 tests are usually about the confusion between identity and equality in Python. Use ==/!= to compare str, bytes, and int literals is the classic case. These are areas where a == b is True but a is b is False (or vice versa). Python >= 3.8 will raise SyntaxWarnings on these instances.
• F7 tests logic errors and syntax errors in type hints
• F82 tests are almost always undefined names which are usually a sign of a typo, missing imports, or code that has not been ported to Python 3. These also would be compile-time errors in a compiled language but in Python, a NameError is raised which will halt/crash the script on the user.