Energy-based Latent Aligner for Incremental Learning
Accepted to CVPR 2022
We illustrate an Incremental Learning model trained on a continuum of tasks in the top part of the figure. While learning the current task , the latent representation of Task
data gets disturbed, as shown by red arrows. ELI learns an energy manifold, and uses it to counteract this inherent representational shift, as illustrated by green arrows, thereby alleviating forgetting.
Overview
In this work, we propose ELI: Energy-based Latent Aligner for Incremental Learning, which:
- Learns an energy manifold for the latent representations such that previous task latents will have low energy and the current task latents have high energy values.
- This learned manifold is used to counter the representational shift that happens during incremental learning.
The implicit regularization that is offered by our proposed methodology can be used as a plug-and-play module in existing incremental learning methodologies for classification and object-detection.
Toy Experiment
A key hypothesis that we base our methodology is that while learning a new task, the latent representations will get disturbed, which will in-turn cause catastrophic forgetting of the previous task, and that an energy manifold can be used to align these latents, such that it alleviates forgetting.
Here, we illustrate a proof-of-concept that our hypothesis is indeed true. We consider a two task experiment on MNIST, where each task contains a subset of classes: = {0, 1, 2, 3, 4},
= {5, 6, 7, 8, 9}.
After learning the second task, the accuracy on test set drops to 20.88%, while experimenting with a 32 dimensional latent space. The latent aligner in ELI provides 62.56% improvement in test accuracy to 83.44%. The visualization of a 512 dimensional latent space after learning
in sub-figure (c), indeed shows cluttering due to representational shift. ELI is able to align the latents as shown in sub-figure (d), which alleviates the drop in accuracy from 89.14% to 99.04%.
The code for these toy experiments are in:
Implicitly Recognizing and Aligning Important Latents
latents.mp4
Each row shows how
latent dimension is updated by ELI. We see that different dimensions have different degrees of change, which is implicitly decided by our energy-based model.
Classification and Detection Experiments
Code and models for the classification and object detection experiments are inside the respective folders:
Each of these are independent repositories. Please consider them separate.
Citation
If you find our research useful, please consider citing us:
@inproceedings{joseph2022Energy,
title={Energy-based Latent Aligner for Incremental Learning},
author={Joseph, KJ and Khan, Salman and Khan, Fahad Shahbaz and Anwar, Rao Muhammad and Balasubramanian, Vineeth},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2022}
}
78 Dec 2, 2022
33 Nov 27, 2022
![[TPAMI 2021] iOD: Incremental Object Detection via Meta-Learning](https://user-images.githubusercontent.com/4231550/138396577-bdef2d95-5f00-47c4-bf90-927d7231f090.png)
66 Jan 4, 2023
258 Dec 31, 2022
57 Nov 17, 2022
41 Nov 22, 2022
42 Nov 9, 2022
![[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing](https://github.com/genforce/interfacegan/raw/master/docs/assets/teaser.jpg)
1.3k Dec 29, 2022
164 Dec 5, 2022
34 Oct 26, 2022
87 Jan 8, 2023
64 Jan 7, 2023
117 Nov 27, 2022
12 Nov 11, 2022
32 Nov 25, 2022
3 Aug 20, 2022
37 Nov 27, 2022
40 Dec 23, 2022
108 Jan 1, 2023