Code release for The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification (TIP 2020)

Overview

The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification

Code release for The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification (TIP 2020) DOI

Changelog

  • 2020/09/14 update the code: CUB-200-2011_ResNet18.py Training with ResNet18 (TRAINED FROM SCRATCH).
  • 2020/04/19 add the hyper-parameter fine-tune results.
  • 2020/04/18 clean the code for better understanding.

Dataset

CUB-200-2011

Requirements

  • python 3.6
  • PyTorch 1.2.0
  • torchvision

Training

  • Download datasets
  • Train: python CUB-200-2011.py, the alpha and beta are the hyper-parameters of the MC-Loss
  • Description : PyTorch CUB-200-2011 Training with VGG16 (TRAINED FROM SCRATCH).

Hyper-parameter

Loss = ce_loss + alpha_1 * L_dis + beta_1 * L_div
Hyper-parameter_1 Hyper-parameter_2 The figure is plot by NNI.

Other versions

Other unofficial implements can be found in the following:

  • Kurumi233: This repo integrate the MC-Loss into a class. code
  • darcula1993: This repo implement the tf version of the MC-Loss. code

Citation

If you find this paper useful in your research, please consider citing:

@ARTICLE{9005389, 
author={D. {Chang} and Y. {Ding} and J. {Xie} and A. K. {Bhunia} and X. {Li} and Z. {Ma} and M. {Wu} and J. {Guo} and Y. {Song}}, 
journal={IEEE Transactions on Image Processing}, 
title={The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification}, 
year={2020}, volume={29}, number={}, pages={4683-4695}, 
doi={10.1109/TIP.2020.2973812}, 
ISSN={1941-0042}, 
month={},} 

Contact

Thanks for your attention! If you have any suggestion or question, you can leave a message here or contact us directly:

Comments
  • About multi-gpu  and the details of training

    About multi-gpu and the details of training

    I can't run this code when I want to use multi-GPU. I only change os.environ["CUDA_VISIBLE_DEVICES"] = "2,3" Is there any question?DO I need change another code?

    enhancement 
    opened by morning4346 16
  • Interfacing with GradCAM?

    Interfacing with GradCAM?

    Is it possible to update the documentation or provide some instructions on how we can reproduce GradCAM like visualizations with the code that is provided?

    Good issue 
    opened by AND2797 9
  • 大佬,我又来了,这次是关于判别性区域的问题

    大佬,我又来了,这次是关于判别性区域的问题

    Screenshot from 2020-09-14 17-23-27 大佬,我在用预训练的resent50训练mcloss网络之后,得到了最终模型,我用的是你在issues中推荐的复现链接,acc为86.6,可能是batch只有32的原因。 不过我的问题不在于此。如上图所示我在可视化特征图时,发现将近有一半的特征图把注意力放在了背景信息上。我的可视化代码应该没有任何问题。请问,这种问题你有尝试过解决吗?还是说我的训练结果有问题,亦或是代码有问题?

     heat = he.data.cpu().numpy()    # heat为最后一层卷积的输出特征图
     print('heat',heat.shape)
     heat = np.squeeze(heat, 0)
     heat = heat[52*10+9,:]
     # heat = heat[2:3, :]
     heatmap = np.maximum(heat, 0)  # heatmap与0比较,取其大者
     # heatmap = np.mean(heatmap, axis=0)
     heatmap /= np.max(heatmap)
     #plt.matshow(heatmap)
     #plt.show()
    
     # 用cv2加载原始图像
     img = cv2.imread('/home/jim/data/FGVC/CUB_200_2011/images/053.Western_Grebe/Western_Grebe_0007_36074.jpg')
     heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))  # 将热力图的大小调整为与原始图像相同
     heatmap = np.uint8(255 * heatmap)  # 将热力图转换为RGB格式
     print(type(heatmap))
    
     heatmap = cv2.applyColorMap(heatmap, cv2.cv2.COLORMAP_HSV )  # 将热力图应用于原始图像
     heat_img = cv2.addWeighted(img, 1, heatmap, 0.5, 0)
     superimposed_img = heatmap * 0.3 + img * 0.7  # 这里的0.4是热力图强度因子
     cv2.imwrite('./self/heat_9.jpg', heat_img)  # 将图像保存到硬盘
     cv2.imshow('heat_11.jpg', heat_img)
     cv2.waitKey()
    
    opened by Javacr 8
  • Can this idea transfer to the recognition task that has a lot of category?

    Can this idea transfer to the recognition task that has a lot of category?

    Hi ! I have reproduced your paper and think it's a wonderful idea to make network learning diversity feature. I have a question that there is a small amont of categories in fined-grained recognition(eg. 200 classes), but how to deal with the task that has a big amount of categories(eg. 5000 classes)? And whether it work when i make a channel to represent more than one category?

    opened by lao-ling-jie 7
  • Codes for training with ResNet18

    Codes for training with ResNet18

    Hi authors: Could you release codes for training with ResNet18 (trained from scratch)? I find it is hard to obtain the reported scores (45.7 with CE loss) training with the hyperparameters provided in your paper. Maybe I miss some critical issues.

    opened by ChunyunShen 7
  • About Channel-Wise Attention(CWA) in the code.

    About Channel-Wise Attention(CWA) in the code.

    Great work!But I have little question about CWA.In the origianl paper, I see M_i = diag(Mask_i), where diag is putting a vector on the princial diagonal of a diagonal matrix.But in the code below:

    foo = [1] * 2 + [0] *  1
    bar = []
    for i in range(200):
        random.shuffle(foo)
        bar += foo
    bar = [bar for i in range(nb_batch)]
    

    I think bar is not a diagonal matrix. Please point out my problem if I misunderstood the operation here.Thanks a lot.

    Good issue 
    opened by Hoodythree 5
  • Does mc-loss need more training epoches?

    Does mc-loss need more training epoches?

    Hi, I added mc-loss to mnasnet (https://arxiv.org/abs/1807.11626v3) network and train it on a custom fine-grained dataset. The total epoches is 15, initial LR is 2e-2, and final LR is 1e-5 using cosine LR scheduler. But the validation accuracy at epoch 15 is 0.62, while the original mnasnet implementation reaches 0.78 val accuracy at epoch 15.

    The training epoches in the paper is 300, is this the cause? (my GPU is slow, so I want to experiment for less epoches to determine whether mc-loss performs good on this dataset)

    Thanks for your great work!

    opened by gbstack 4
  • Question about Ldis in code

    Question about Ldis in code

    Wonderful work! But i have some question about Ldis. In your paper, i see specific channels belong to a class(eg. channel1 to channel3 is belong to class 1 ), but how to realize in code? you don't use targets when calculating the mask

    opened by lao-ling-jie 4
  • Reproducing the results

    Reproducing the results

    Hello,

    thanks again for your interesting work. I however, struggle to reproduce your reported results on CUB200 with the Resnet50 backbone (87.3 %). If I understand https://github.com/Kurumi233/Mutual-Channel-Loss/issues/1 correctly, then you used entirely different hyperparameters than those mentioned in the paper? I also saw that you deviated (as mentioned here:https://github.com/PRIS-CV/Mutual-Channel-Loss/issues/21) from the formulas of the paper, which impacts the hyperparameters. Could you point me to the correct combination of hyperparameters and code / formulas to reproduce your results? I appreciate your help.

    Greetings

    opened by Gnabe 2
  • About validation set

    About validation set

    Hi,

    I recently did some research on fine-grained classification, and I found that you mentioned you didn't divide a validation set on CUB_200_2011 in your paper. Does it mean that when I do some experiments on CUB_200_2011, I can just take the test set as the valid set and I can take the best validation performance as my final result?

    Hope to get help from you, since I can't find descriptions about the validation set in other existing papers.

    Thank you.

    Kind regards.

    opened by WindVChen 2
  • About  λ(bata_1)

    About λ(bata_1)

    Great work!But I have a little question about λ. In your code: loss = ce_loss + args["alpha_1"] * MC_loss[0] + args["beta_1"] * MC_loss[1] And in your paper: image

    So,I should set beta_1 as -10 in code? Please point out my problem if I misunderstood the operation here.Thanks a lot.

    opened by DeepDetector 2
Owner
PRIS-CV: Computer Vision Group
CV Group , PRIS lab, School of Artificial Intelligence, BUPT
PRIS-CV: Computer Vision Group
Saeed Lotfi 28 Dec 12, 2022
PyTorch implementation of Weak-shot Fine-grained Classification via Similarity Transfer

SimTrans-Weak-Shot-Classification This repository contains the official PyTorch implementation of the following paper: Weak-shot Fine-grained Classifi

BCMI 60 Dec 2, 2022
Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification

Fine-grainedImageClassification Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification We trained model here: lin

ZhenchaoTang 14 Oct 21, 2022
A Novel Plug-in Module for Fine-grained Visual Classification

Pytorch implementation for A Novel Plug-in Module for Fine-Grained Visual Classification. fine-grained visual classification task.

ChouPoYung 109 Dec 20, 2022
Code for the TIP 2021 Paper "Salient Object Detection with Purificatory Mechanism and Structural Similarity Loss"

PurNet Project for the TIP 2021 Paper "Salient Object Detection with Purificatory Mechanism and Structural Similarity Loss" Abstract Image-based salie

Jinming Su 4 Aug 25, 2022
Official pytorch code for SSC-GAN: Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation(ICCV 2021)

SSC-GAN_repo Pytorch implementation for 'Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation'.PDF SSC-GAN:Sem

tyty 4 Aug 28, 2022
Fine-grained Control of Image Caption Generation with Abstract Scene Graphs

Faster R-CNN pretrained on VisualGenome This repository modifies maskrcnn-benchmark for object detection and attribute prediction on VisualGenome data

Shizhe Chen 7 Apr 20, 2021
The code release of paper 'Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization' NIPS 2020.

Domain Generalization for Medical Imaging Classification with Linear Dependency Regularization The code release of paper 'Domain Generalization for Me

Yufei Wang 56 Dec 28, 2022
Code and data of the Fine-Grained R2R Dataset proposed in paper Sub-Instruction Aware Vision-and-Language Navigation

Fine-Grained R2R Code and data of the Fine-Grained R2R Dataset proposed in the EMNLP2020 paper Sub-Instruction Aware Vision-and-Language Navigation. C

YicongHong 34 Nov 15, 2022
Code for Talk-to-Edit (ICCV2021). Paper: Talk-to-Edit: Fine-Grained Facial Editing via Dialog.

Talk-to-Edit (ICCV2021) This repository contains the implementation of the following paper: Talk-to-Edit: Fine-Grained Facial Editing via Dialog Yumin

Yuming Jiang 221 Jan 7, 2023
FIRA: Fine-Grained Graph-Based Code Change Representation for Automated Commit Message Generation

FIRA is a learning-based commit message generation approach, which first represents code changes via fine-grained graphs and then learns to generate commit messages automatically.

Van 21 Dec 30, 2022
PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020).

NHDRRNet-PyTorch This is the PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020). 0. Differences between Original Paper and

Yutong Zhang 1 Mar 1, 2022
This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

TransFG: A Transformer Architecture for Fine-grained Recognition Official PyTorch code for the paper: TransFG: A Transformer Architecture for Fine-gra

Ju He 307 Jan 3, 2023
WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose

WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose Yijun Zhou and James Gregson - BMVC2020 Abstract: We present an end-to-end head-pos

null 368 Dec 26, 2022
The coda and data for "Measuring Fine-Grained Domain Relevance of Terms: A Hierarchical Core-Fringe Approach" (ACL '21)

We propose a hierarchical core-fringe learning framework to measure fine-grained domain relevance of terms – the degree that a term is relevant to a broad (e.g., computer science) or narrow (e.g., deep learning) domain.

Jie Huang 14 Oct 21, 2022
The implementation of CVPR2021 paper Temporal Query Networks for Fine-grained Video Understanding, by Chuhan Zhang, Ankush Gupta and Andrew Zisserman.

Temporal Query Networks for Fine-grained Video Understanding ?? This repository contains the implementation of CVPR2021 paper Temporal_Query_Networks

null 55 Dec 21, 2022