Official repository of the paper 'Essentials for Class Incremental Learning'

Overview

Essentials for Class Incremental Learning

Official repository of the paper 'Essentials for Class Incremental Learning'

This Pytorch repository contains the code for our work Essentials for Class Incremental Learning.

This work presents a straightforward class-incrmental learning system that focuses on the essential components and already exceeds the state of the art without integrating sophisticated modules.

Requirements

To install requirements:

pip install -r requirements.txt

Training and Evaluation (CIFAR-100, ImageNet-100, ImageNet-1k)

Following scripts contain both training and evaluation codes. Model is evaluated after each phase in class-IL.

with Knowledge-distillation (KD)

To train the base CCIL model:

bash ./scripts/run_cifar.sh
bash ./scripts/run_imagenet100.sh
bash ./scripts/run_imagenet1k.sh

To train CCIL + Self-distillation

bash ./scripts/run_cifar_w_sd.sh
bash ./scripts/run_imagenet100_w_sd.sh
bash ./scripts/run_imagenet1k_w_sd.sh

Results (CIFAR-100)

Model name Avg Acc (5 iTasks) Avg Acc (10 iTasks)
CCIL 66.44 64.86
CCIL + SD 67.17 65.86

Results (ImageNet-100)

Model name Avg Acc (5 iTasks) Avg Acc (10 iTasks)
CCIL 77.99 75.99
CCIL + SD 79.44 76.77

Results (ImageNet)

Model name Avg Acc (5 iTasks) Avg Acc (10 iTasks)
CCIL 67.53 65.61
CCIL + SD 68.04 66.25

List of Arguments

  • Distillation Methods

    • Knowledge Distillation (--kd, --w-kd X), X is the weightage for KD loss, default=1.0
    • Representation Distillation (--rd, --w-rd X), X is the weightage for cos-RD loss, default=0.05
    • Contrastive Representation Distillation (--nce, --w-nce X), only valid for CIFAR-100, X is the weightage of NCE loss
  • Regularization for the first task

    • Self-distillation (--num-sd X, --epochs-sd Y), X is number of generations, Y is number of self-distillation epochs
    • Mixup (--mixup, --mixup-alpha X), X is mixup alpha value, default=0.1
    • Heavy Augmentation (--aug)
    • Label Smoothing (--label-smoothing, --smoothing-alpha X), X is a alpha value, default=0.1
  • Incremental class setting

    • No. of base classes (--start-classes 50)
    • 5-phases (--new-classes 10)
    • 10-phases (--new-classes 5)
  • Cosine learning rate decay (--cosine)

  • Save and Load

    • Experiment Name (--exp-name X)
    • Save checkpoints (--save)
    • Resume checkpoints (--resume, --resume-path X), only to resume from first snapshot

Citation

@article{ccil_mittal,
    Author = {Sudhanshu Mittal and Silvio Galesso and Thomas Brox},
    Title = {Essentials for Class Incremental Learning},
    journal = {arXiv preprint arXiv:2102.09517},
    Year = {2021},
}
Comments
  • someting wrong when create exemplar_set?

    someting wrong when create exemplar_set?

    ==> Current Class: [50, 51, 52, 53, 54, 55, 56, 57, 58, 59] ==> Building model.. in_features: 512 out_features: 50 current net output dim: 60 old net output dim: 50 [50, 51, 52, 53, 54, 55, 56, 57, 58, 59] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] Constructing exemplar set for class-50... exemplar set shape: 33 Done Constructing exemplar set for class-51... exemplar set shape: 33 Done Constructing exemplar set for class-52... exemplar set shape: 33 Done Constructing exemplar set for class-53... exemplar set shape: 33 Done Constructing exemplar set for class-54... exemplar set shape: 33 Done Constructing exemplar set for class-55... exemplar set shape: 33 Done Constructing exemplar set for class-56... exemplar set shape: 33 Done Constructing exemplar set for class-57... exemplar set shape: 33 Done Constructing exemplar set for class-58... exemplar set shape: 33 Done Constructing exemplar set for class-59... exemplar set shape: 33 Done start self-distillation for original model..... setting optimizer and scheduler................. Traceback (most recent call last): File "main_imagenet.py", line 472, in train(model=net, old_model=old_net, epoch=args.epochs, optimizer=optimizer, scheduler=scheduler, lamda=args.lamda, train_loader=trainLoader, use_sd=False, checkPoint=50) File "main_imagenet.py", line 151, in train exemplar_set = ExemplarDataset(exemplar_sets, transform=transform_ori) File "/home/ubuntu/Desktop/Alex/IL/essentials_for_CIL/data/data_loader_imagenet.py", line 17, in init self.data = np.concatenate(data, axis=0) File "<array_function internals>", line 6, in concatenate ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 4 dimension(s) and the array at index 1 has 1 dimension(s)

    opened by alexHxun 1
  • How to get the results of CCIL+H-Aug ?

    How to get the results of CCIL+H-Aug ?

    I expect to get the result of CCIL+H-Aug 71.66, but I get 68.25.

    I conduct the following command

    python main_cifar.py  \
            --new-classes 10 \
            --start-classes 50 \
            --cosine \
            --kd \
            --w-kd 1 \
            --epochs 120 \
            --exp-name 'kd_ep120_50_5' \
            --save \
            --num-sd 0\
            --aug  
    

    Results: [79.56, 74.01666666666667, 69.98571428571428, 65.1375, 61.63333333333333, 59.16] Avg accuracy: 68.24886904761904

    Do I miss some details?

    Expect your reply. Thanks

    opened by wusuoweima 0
  • Which version of ImageNet is used?

    Which version of ImageNet is used?

    Hi Sudhanshu,

    Thank you for your nice work and codebase.

    If possible, can you please share the version of ImageNet 1000 that you have used? Is it ImageNet 2017? If you have the download link handy, it would be very helpful if you could share them.

    Thanks!

    opened by JosephKJ 0
  • About icarl_construct_exemplar_set function

    About icarl_construct_exemplar_set function

    Hi, I think the following codes seem not proper: https://github.com/sud0301/essentials_for_CIL/blob/c7eb378c822792977d96daa5dd98c0202263ec44/main_cifar.py#L321-L337 After debugging I find that when features[idx, :] = 0.0 implemented, exemplar_features is also set to 0, thus S is always full of 0 at any time, making it meaningless. I am not sure whether it is a common problem or it is triggered due to my incompatible version of packages. Besides, what if the random saving strategy get the same index? Is it right the purpose of setting the corresponding feature to 0, so that in the next phase the repetitive sample has a large distance and will be deleted? But still, what if class_mean itself is close to 0?

    opened by superduduguan 1
  • About the way you read image

    About the way you read image

    In 'main_imagenet.py' , line 321:

    x = Variable(transform(Image.fromarray(img))).cuda()

    You use Image.fromarray(img) to load a array type image object. However, channel order in img is BGR (opencv) while the channel order in Image is RGB, which means this operation will confuse the channel order.

    opened by LionRoarRoar 0
Owner
null
The code repository for "PyCIL: A Python Toolbox for Class-Incremental Learning" in PyTorch.

PyCIL: A Python Toolbox for Class-Incremental Learning Introduction • Methods Reproduced • Reproduced Results • How To Use • License • Acknowledgement

Fu-Yun Wang 258 Dec 31, 2022
The official PyTorch code for 'DER: Dynamically Expandable Representation for Class Incremental Learning' accepted by CVPR2021

DER.ClassIL.Pytorch This repo is the official implementation of DER: Dynamically Expandable Representation for Class Incremental Learning (CVPR 2021)

rhyssiyan 108 Jan 1, 2023
Official Pytorch implementation of Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (ICLR 2022)

The Official Implementation of CLIB (Continual Learning for i-Blurry) Online Continual Learning on Class Incremental Blurry Task Configuration with An

NAVER AI 34 Oct 26, 2022
Implementation of the paper "Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning"

Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning This is the implementation of the paper "Self-Promoted Prototype Refinement

Kai Zhu 78 Dec 2, 2022
This is the formal code implementation of the CVPR 2022 paper 'Federated Class Incremental Learning'.

Official Pytorch Implementation for GLFC [CVPR-2022] Federated Class-Incremental Learning This is the official implementation code of our paper "Feder

Race Wang 57 Dec 27, 2022
Code for "Learning Structural Edits via Incremental Tree Transformations" (ICLR'21)

Learning Structural Edits via Incremental Tree Transformations Code for "Learning Structural Edits via Incremental Tree Transformations" (ICLR'21) 1.

NeuLab 40 Dec 23, 2022
[TPAMI 2021] iOD: Incremental Object Detection via Meta-Learning

Incremental Object Detection via Meta-Learning To appear in an upcoming issue of the IEEE Transactions on Pattern Analysis and Machine Intelligence (T

Joseph K J 66 Jan 4, 2023
Reinforcement-learning - Repository of the class assignment questions for the course on reinforcement learning

DSE 314/614: Reinforcement Learning This repository containing reinforcement lea

Manav Mishra 4 Apr 15, 2022
Sound and Cost-effective Fuzzing of Stripped Binaries by Incremental and Stochastic Rewriting

StochFuzz: A New Solution for Binary-only Fuzzing StochFuzz is a (probabilistically) sound and cost-effective fuzzing technique for stripped binaries.

Zhuo Zhang 164 Dec 5, 2022
🔥 TensorFlow Code for technical report: "YOLOv3: An Incremental Improvement"

?? Are you looking for a new YOLOv3 implemented by TF2.0 ? If you hate the fucking tensorflow1.x very much, no worries! I have implemented a new YOLOv

null 3.6k Dec 26, 2022
Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding (CVPR2022)

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding by Qiaole Dong*, Chenjie Cao*, Yanwei Fu Paper and Supple

Qiaole Dong 190 Dec 27, 2022
Official implementation for the paper: "Multi-label Classification with Partial Annotations using Class-aware Selective Loss"

Multi-label Classification with Partial Annotations using Class-aware Selective Loss Paper | Pretrained models Official PyTorch Implementation Emanuel

null 99 Dec 27, 2022
Official implementation for CVPR 2021 paper: Adaptive Class Suppression Loss for Long-Tail Object Detection

Adaptive Class Suppression Loss for Long-Tail Object Detection This repo is the official implementation for CVPR 2021 paper: Adaptive Class Suppressio

CASIA-IVA-Lab 67 Dec 4, 2022
Official codes: Self-Supervised Learning by Estimating Twin Class Distribution

TWIST: Self-Supervised Learning by Estimating Twin Class Distributions Codes and pretrained models for TWIST: @article{wang2021self, title={Self-Sup

Bytedance Inc. 85 Dec 15, 2022
The implementation of the algorithm in the paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020.

DS3L This is the code for paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020. Setups The code is implem

Guolz 36 Oct 19, 2022
Tensorflow 2 implementation of the paper: Learning and Evaluating Representations for Deep One-class Classification published at ICLR 2021

Deep Representation One-class Classification (DROC). This is not an officially supported Google product. Tensorflow 2 implementation of the paper: Lea

Google Research 137 Dec 23, 2022
Official implementation of "A Unified Objective for Novel Class Discovery", ICCV2021 (Oral)

A Unified Objective for Novel Class Discovery This is the official repository for the paper: A Unified Objective for Novel Class Discovery Enrico Fini

Enrico Fini 118 Dec 26, 2022
Official implementation of Generalized Data Weighting via Class-level Gradient Manipulation (NeurIPS 2021).

Generalized Data Weighting via Class-level Gradient Manipulation This repository is the official implementation of Generalized Data Weighting via Clas

null 9 Nov 3, 2021
The first machine learning framework that encourages learning ML concepts instead of memorizing class functions.

SeaLion is designed to teach today's aspiring ml-engineers the popular machine learning concepts of today in a way that gives both intuition and ways of application. We do this through concise algorithms that do the job in the least jargon possible and examples to guide you through every step of the way.

Anish 324 Dec 27, 2022