Thanks for your great job! I have browsed your code and I found that the data is read in through the ImageDataset class, but it does not seem to fit the original format of the CUB dataset. Did you change the format of the original CUB dataset when you performed your experiments, and if so, can you please tell me how you did it?

Hi, I am running a test on your repo with Stanford Dog Dataset which has 120 species. The model trained really well, but I am a little confused with your inference pipeline. I just want to run a inference on a single image so I am referring to your eval.py and plot_heat.py at the moment.

Your eval.py seems to be calling SwinVit12, but plot_heat.py seems to be calling SwinVit12_demo. Are there difference between the two?

Just tried running eval.py and I am getting:

RuntimeError: Error(s) in loading state_dict for SwinVit12:
        size mismatch for gcn.adj1: copying a param with shape torch.Size([85, 85]) from checkpoint, the shape in current model is torch.Size([15, 15]).
        size mismatch for gcn.pool1.weight: copying a param with shape torch.Size([85, 2720]) from checkpoint, the shape in current model is torch.Size([15, 480]).
        size mismatch for gcn.pool1.bias: copying a param with shape torch.Size([85]) from checkpoint, the shape in current model is torch.Size([15]).
        size mismatch for gcn.pool4.weight: copying a param with shape torch.Size([1, 85]) from checkpoint, the shape in current model is torch.Size([1, 15]).

Seems like something is not configured properly on my end.

Hi, Thanks for your excellent work, i have a question, i just find the pre-training model Swin_t with pre-training on i1k and resolution 224, can you provide the link to download the pre-training model of swin_t in the paper?

Why is the output of line 331 of models/pim_module/pim_module.py a dict and not a tensor？That is When I test it, the err is I use swin-T as the backbone. Can you help me?

Thanks for your code, it is really a nice work! However, I found multiple troubles when adapting the code to my own data. I have already: 1) changed the class number in the config.py and set the args in CMD; 2) the inputted image size has been changed following former issues in this Github page.

However, after changing these two aspects, the results are still misleading and confusing. So, may I ask, if using the model on one's own dataset, how many issues do I need to change?

I checked the code carefully. You write the loss calculation in the model. After initializing the model, the model needs to be sent to the label. There is no code design for infer, and after the end-to-end training, a series of post-processing is used. , including: splicing features, fusion features, etc., and then classify them separately, I want to know if I don't know gt, how to choose the best result? Is this code just written to brush the list? Where is the logic of the actual application?

This is a great job, but I ran on my own dataset and encountered the following errors.

Start Training 3 Epoch..0%..10%..20% Start Evaluating 1 Epoch Start Training 2 Epoch Start Training 3 Epoch Traceback (most recent call last): File "main.py", line 297, in main(args, tlogger) File "main.py", line 249, in main train(args, epoch, model, scaler, amp_context, optimizer, schedule, train_loader) File "main.py", line 136, in train outs = model(datas) File "/DATA/sgwei/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/DATA/sgwei/code/Fine_grained_PIM/models/pim_module/pim_module.py", line 404, in forward x = self.forward_backbone(x) File "/DATA/sgwei/code/Fine_grained_PIM/models/pim_module/pim_module.py", line 383, in forward_backbone return self.backbone(x) File "/DATA/sgwei/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/fx/graph_module.py", line 616, in wrapped_call raise e.with_traceback(None) RecursionError: maximum recursion depth exceeded while calling a Python object

and this is my configs

batch_size 16 c "./configs/classification_vit.yaml" data_size 448 device "cuda:0" eval_freq 10 exp_name "T3000" fpn_size 512 lambda_b 0.5 lambda_c 1 lambda_n 5 lambda_s 0 log_freq 100 max_epochs 20 max_lr 0.0005 model_name "vit" num_classes 5 num_selects layer1 512 layer2 256 layer3 128 layer4 64 num_workers 8 optimizer "SGD" pretrained desc null value null project_name "pim_classificationpip" save_dir "./records/pim_classificationpip/T3000/" train_root "/DATA/sgwei/Datasets/DataBase_v2/train/" update_freq 2 use_amp true use_combiner true use_fpn true use_selection true use_wandb true val_root "/DATA/sgwei/Datasets/MultiLesionClassify_DataBase_v2/val/" wandb_entity "sgwei" warmup_batchs 800 wdecay 0.0005

Dear author, I saw in your code said "Vit model input can transform 224 to another, we use linear", but I do not know how to use it. I tried to use 384*384 as my input size directly, but it shows me a tensor cannot match issues, so I have to resize my input data, but there are same issues when I detect my test data, so can I know how to use transform 224 to another, or I just need to add a linear layer before I input my datasets?

Thanks

while run

python main.py --c ./configs/eval.yaml

your code will be wrong

I solved this problem and added eval.yaml, when verifying the result, I need to put in the yaml：

pretrained: your model weight path

Then：

python main.py --c ./configs/eval.yaml

will get the correct result

Thank you so much for the beautiful code. I'm trying to use your pretrained model best.pt on NABirds dataset.

First, I set the PATH to the pretrained model in NABirds_SwinT.yaml then I run:

python heat.py --c ./configs/NABirds_SwinT.yaml --img ./vis/001.jpg --save_img ./vis/001/

But I get errors:

Building... Traceback (most recent call last): File "C:\Users\xxxxxx\Desktopxxxxxx\heat.py", line 100, in model.load_state_dict(checkpoint['model']) File "C:\Users\xxxxxx\anaconda3\envs\xxxxxxx\lib\site-packages\torch\nn\modules\module.py", line 1667, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for PluginMoodel: Unexpected key(s) in state_dict: "combiner.conv_qk1.weight", "combiner.conv_qk1.bias". size mismatch for combiner.adj1: copying a param with shape torch.Size([85, 85]) from checkpoint, the shape in current model is torch.Size([15, 15]). size mismatch for combiner.param_pool0.weight: copying a param with shape torch.Size([85, 2720]) from checkpoint, the shape in current model is torch.Size([15, 480]). size mismatch for combiner.param_pool0.bias: copying a param with shape torch.Size([85]) from checkpoint, the shape in current model is torch.Size([15]). size mismatch for combiner.param_pool1.weight: copying a param with shape torch.Size([1, 85]) from checkpoint, the shape in current model is torch.Size([1, 15]).

It seems that your best,pt model is not using default SwinT model, the num_selects is not matching And there are unexpected keys in your best.pt model: "combiner.conv_qk1.weight", "combiner.conv_qk1.bias"

I wish to know what modification I should make to load your pretrained best.pt.

When I select efficentnet for training I get the following error, only swin-transformer does not report it Can you help me?

Start Training 1 EpochTraceback (most recent call last): File "D:/hxy/FGVC-PIM-master/main.py", line 301, in main(args, tlogger) File "D:/hxy/FGVC-PIM-master/main.py", line 253, in main train(args, epoch, model, scaler, amp_context, optimizer, schedule, train_loader) File "D:/hxy/FGVC-PIM-master/main.py", line 140, in train outs = model(datas) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\data_parallel.py", line 168, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\data_parallel.py", line 178, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\parallel_apply.py", line 86, in parallel_apply output.reraise() File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch_utils.py", line 457, in reraise raise exception RuntimeError: Caught RuntimeError in replica 0 on device 0. Original Traceback (most recent call last): File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\parallel_apply.py", line 61, in _worker output = module(*input, **kwargs) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "D:\hxy\FGVC-PIM-master\models\pim_module\pim_module.py", line 414, in forward comb_outs = self.combiner(selects) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "D:\hxy\FGVC-PIM-master\models\pim_module\pim_module.py", line 81, in forward hs = self.param_pool0(hs) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "C:\Users\mj\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\linear.py", line 103, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 shapes cannot be multiplied (6144x1456 and 2720x85)