Implementation of the paper "Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning"

Kai Zhu

Last update: Dec 2, 2022

Related tags

Deep Learning SPPR

Overview

Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning

This is the implementation of the paper "Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning" (accepted to CVPR2021).

For more information, check out the paper on [arXiv].

Requirements

Python 3.8
PyTorch 1.8.1 (>1.1.0)
cuda 11.2

Preparing Few-Shot Class-Incremental Learning Datasets

Download following datasets:

1. CIFAR-100

Automatically downloaded on torchvision.

2. MiniImageNet

(1) Download MiniImageNet train/test images[github], and prepare related datasets according to [TOPIC].

(2) or Download processed data from our Google Drive: [mini-imagenet.zip], (and locate the entire folder under datasets/ directory).

3. CUB200

(1) Download CUB200 train/test images, and prepare related datasets according to [TOPIC]:
wget http://www.vision.caltech.edu/visipedia-data/CUB-200-2011/CUB_200_2011.tgz
(2) or Download processed data from our Google Drive: [cub.zip], (and locate the entire folder under datasets/ directory).

Create a directory '../datasets' for the above three datasets and appropriately place each dataset to have following directory structure:

../                                                        # parent directory
├── ./                                           # current (project) directory
│   ├── log/                              # (dir.) running log
│   ├── pre/                              # (dir.) trained models for test.
│   ├── utils/                            # (dir.) implementation of paper 
│   ├── README.md                          # intstruction for reproduction
│   ├── test.sh                          # bash for testing.
│   ├── train.py                        # code for training model
│   └── train.sh                        # bash for training model
└── datasets/
    ├── CIFAR100/                      # CIFAR100 devkit
    ├── mini-imagenet/           
    │   ├── train/                         # (dir.) training images (from Google Drive)
    │   ├── test/                           # (dir.) testing images (from Google Drive)
    │   └── ..some csv files..
    └── cub/                                   # (dir.) contains 200 object classes
        ├── train/                             # (dir.) training images (from Google Drive)
        └── test/                               # (dir.) testing images (from Google Drive)

Training

Choose apporopriate lines in train.sh file.
sh train.sh
'--base_epochs' can be modified to control the initial accuracy ('Our' vs 'Our*' in our paper).

Training takes approx. several hours until convergence (trained with one 2080 Ti or 3090 GPUs).

Testing

1. Download pretrained models to the 'pre' folder.

Pretrained models are available on our [Google Drive].

2. Test

Choose apporopriate lines in train.sh file.
sh test.sh 

Main Results

The experimental results with 'test.sh 'for three datasets are shown below.

1. CIFAR-100

Model	1	2	3	4	5	6	7	8	9
iCaRL	64.10	53.28	41.69	34.13	27.93	25.06	20.41	15.48	13.73
TOPIC	64.10	56.03	47.89	42.99	38.02	34.60	31.67	28.35	25.86
Ours	63.97	65.86	61.31	57.6	53.39	50.93	48.27	45.36	43.32

2. MiniImageNet

Model	1	2	3	4	5	6	7	8	9
iCaRL	61.31	46.32	42.94	37.63	30.49	24.00	20.89	18.80	17.21
TOPIC	61.31	45.58	43.77	37.19	32.38	29.67	26.44	25.18	21.80
Ours	61.45	63.80	59.53	55.53	52.50	49.60	46.69	43.79	41.92

3. CUB200

Model	1	2	3	4	5	6	7	8	9	10	11
iCaRL	68.68	52.65	48.61	44.16	36.62	29.52	27.83	26.26	24.01	23.89	21.16
TOPIC	68.68	61.01	55.35	50.01	42.42	39.07	35.47	32.87	30.04	25.91	24.85
Ours	68.05	62.01	57.61	53.67	50.77	46.76	45.43	44.53	41.74	39.93	38.45

The presented results are slightly different from those in the paper, which are the average results of multiple tests.

BibTeX

If you use this code for your research, please consider citing:

@inproceedings{zhu2021self,
  title={Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning},
  author={Zhu, Kai and Cao, Yang and Zhai, Wei and Cheng, Jie and Zha, Zheng-Jun},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={6801--6810},
  year={2021}
}

Comments

Question about the setting

您好! 或许代码中的Few-Shot Class-Incremental Setting与论文中(或是与常用的benchmark, Tao等人提出的)不相符:

在第二个session及以后, 每个session的训练集中仅能包含当前新增类别的数据, 即保证所有session训练集的标签空间不相交, 但是您的代码中my_dataset.py似乎把第二个session及以后的训练数据累加在一起考虑, 这将导致训练时进入额外的数据.

因为没有非常细致地看您的代码, 所以上述也可能是错的, 打搅您了.

非常期望您能给出解答, 谢谢!

Hi! Maybe the Few-Shot Class-Incremental Setting in the code is not consistent with the paper (or with the commonly used benchmark, as proposed by Tao et al.):

In the second session and beyond, each session's training set can only contain data from the currently added class, i.e., the label spaces of all sessions are guaranteed to be disjoint, but your code my_dataset.py seems to consider the second session and subsequent training data cumulatively, which will cause additional data input during training.

Since I haven't looked at your code very carefully, the above may be wrong as well, sorry to bother you.

Looking forward to your answer, thanks!

opened by ZhangYikaii 2
关于对论文"Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning"的问题
您好，非常有幸阅读到您发表在CVPR2021的文章“Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning”。有以下几个问题不太清楚，想向您请教。

在4.2节中提到“Dynamic Relation Projection. To maintain the dependencies [33, 17, 34] of old classes and enhance the discrimination of the new classes, we propose a self-promoted prototype refinement mechanism”，我不太明白这里提到的“增强新类的判别性信息”是从哪里体现出来的，您能解释一下吗，希望得到您的回答。

请问在伪增量学习特征表示后，特征表示器的参数是固定下来，然后用于后续真正的增量学习中吗。
opened by tangdouer 0
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0
code of Dynamic Relation Projection
by /utils/model/my_model.py 1. The paper mention that ft1 and ft2 represents a set of standard convolution block. Might I missed it, but I can't find convolution block for "transform embeddings and the old prototypes". There is only the 1x1 kernal size conv block for getting similarity between query and prototypes. Is attention network be the Dynamic Relation Projection part?

#L113 I'm not sure the code of here, that do tensor multiplication twice, which multi with 6060 and 605 matrix. Is the purpose just replace the new prototypes to prototype list? Or it is part of the Relation Projection Where is the 60*55 relation matrix mention in the paper?

Thanks
opened by lechchu 0
Some questions about the CategoriesSampler
（1）in Trainer.py , line 87 train_sampler = CategoriesSampler(train_dataset.sub_indexes, len(train_loader), self.args.way+3, self.args.shot)

Why collect 3 more classes of samples？？

(2) in lr.py line 88

lenth_per = torch.from_numpy(self.index[c]) way_per = len(lenth_per) shot = torch.randperm(way_per)[:self.shot]

lenth_per is the list of sample indices for category C within the dataset. way_per is the total number of samples in category c. If you want to collect self.shot samples from category c, you should do lenth_per[:self.shot], or random.sample(lenth_per, self.shot), right? And the sample index obtained by the command shot = torch.randperm(num_data_in_one_class)[:self.shot] does not seem to be the true index of the sample。

What is the intention of this operation：c*num_data_in_one_class+shot ？

（3） In the training process of session0, train_loader is used to collect the query set needed for an episode, and the query data within each batch contains the data of all 60 base classes. Meanwhile, the train_fsl_loader is used to sample the data of the support set, and each episode collects three additional classes of data. I don't quite understand what is the connection between the program's episode sampling process and the process described by Random Episode Selection in the paper? Can you provide a more detailed explanation?
opened by chester-w-xie 0
Question about the results of CUB200

Hi, when I use the default settings of the train.sh, the acc of session_0 is 62, approximately. The acc raises to 68 when the base_epoch is increased by 70 to 120, however, the following sessions are worse than those reported in your readme or paper as run in my computer: 68.09, 55.42, 50.00, 48.44, 44.19, 39.57, 37.23, 35.61, 34.92, 33.47, 32.07 reported in the readme: 68.05, 62.01, 57.61, 53.67, 50.77, 46.76, 45.43, 44.53, 41.74, 39.93, 38.45

I have tried several times, similar results still produced. I'm not sure there are some special settings for CUB200 or I have made any running operations by mistake.

opened by huangkainwpu24 1

Implementation of the paper "Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning"

Related tags

Overview

Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning

Requirements