Code to accompany our paper "Continual Learning Through Synaptic Intelligence" ICML 2017

Related tags

Deep Learning pathint
Overview

Continual Learning Through Synaptic Intelligence

This repository contains code to reproduce the key findings of our path integral approach to prevent catastrophic forgetting in continual learning.

Zenke, F.1, Poole, B.1, and Ganguli, S. (2017). Continual Learning Through Synaptic Intelligence. In Proceedings of the 34th International Conference on Machine Learning, D. Precup, and Y.W. Teh, eds. (International Convention Centre, Sydney, Australia: PMLR), pp. 3987–3995.

http://proceedings.mlr.press/v70/zenke17a.html

1) Equal contribution

BibTeX

@InProceedings{pmlr-v70-zenke17a,
title = 	 {Continual Learning Through Synaptic Intelligence},
author = 	 {Friedemann Zenke and Ben Poole and Surya Ganguli},
booktitle = 	 {Proceedings of the 34th International Conference on Machine Learning},
pages = 	 {3987--3995},
year = 	 {2017},
editor = 	 {Doina Precup and Yee Whye Teh},
volume = 	 {70},
series = 	 {Proceedings of Machine Learning Research},
address = 	 {International Convention Centre, Sydney, Australia},
month = 	 {06--11 Aug},
publisher = 	 {PMLR},
pdf = 	 {http://proceedings.mlr.press/v70/zenke17a/zenke17a.pdf},
url = 	 {http://proceedings.mlr.press/v70/zenke17a.html},
}

Requirements

We have tested this maintenance release (v1.1) with the following configuration:

  • Python 3.5.2
  • Jupyter 4.4.0
  • Tensorflow 1.10
  • Keras 2.2.2

Kudos to Mitra (https://github.com/MitraDarja) for making our code conform with Keras 2.2.2!

Earlier releases

For the original release (v1.0) we used the following configuration of the libraries which were available at the time:

  • Python 3.5.2
  • Jupyter 4.3.0
  • Tensorflow 1.2.1
  • Keras 2.0.5

To revert to such a environment we suggest using virtualenv (https://virtualenv.pypa.io):

virtualenv -p python3 env
source env/bin/activate
pip3 install -vI keras==2.0.5
pip3 install jupyter matplotlib numpy tensorflow-gpu tqdm seaborn
You might also like...
Code for the ICML 2021 paper:
Code for the ICML 2021 paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

ViLT Code for the paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision" Install pip install -r requirements.txt pip

Wonjae Kim 922 Jan 1, 2023
Code for the ICML 2021 paper:
Code for the ICML 2021 paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

ViLT Code for the paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision" Install pip install -r requirements.txt pip

Wonjae Kim 922 Jan 1, 2023
Code for ICML 2021 paper: How could Neural Networks understand Programs?
Code for ICML 2021 paper: How could Neural Networks understand Programs?

OSCAR This repository contains the source code of our ICML 2021 paper How could Neural Networks understand Programs?. Environment Run following comman

Dinglan Peng 115 Dec 17, 2022
Official Code for ICML 2021 paper
Official Code for ICML 2021 paper "Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline"

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, Jia Deng Internati

Princeton Vision & Learning Lab 115 Jan 4, 2023
Code for Fold2Seq paper from ICML 2021
Code for Fold2Seq paper from ICML 2021

[ICML2021] Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design Environment file: environment.yml Data and Feat

International Business Machines 43 Dec 4, 2022
Code for the ICML 2021 paper
Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation", Haoxiang Wang, Han Zhao, Bo Li.

Bridging Multi-Task Learning and Meta-Learning Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Trainin

AI Secure 57 Dec 15, 2022
Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun

ARAE Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun https://arxiv.org/abs/1706.04223 Disc

Junbo (Jake) Zhao 399 Jan 2, 2023
Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation".

PixelTransformer Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation". Project Page Installation Please insta

Shubham Tulsiani 24 Dec 17, 2022
Code for paper
Code for paper "Which Training Methods for GANs do actually Converge? (ICML 2018)"

GAN stability This repository contains the experiments in the supplementary material for the paper Which Training Methods for GANs do actually Converg

Lars Mescheder 885 Jan 1, 2023
Comments
  • data augmentation

    data augmentation

    Hey - one quick question. I scanned the code a bit but could not find any data augmentation on cifar 10 / cifar 100. Is that correct? The results reported in the paper on the cifar 10/100 task are achieved by training on the cifar 10 / cifar 100 without data augmentation?
    Many thanks!

    opened by Johswald 2
  • Adds example data to the repo and updates readme

    Adds example data to the repo and updates readme

    This adds example data for comparison to the repository making it unnecessary to run the code for comparison. Moreover, I added a short description to the Readme.md as to how to deal with in incompatibility which arises with our code and newer Keras versions.

    opened by fzenke 0
  • re-compilation error

    re-compilation error

    hi, I'm trying to modify dropout rate per each task recently.

    For that, I should change dropout rates of the layer, and re-compile.

    However, it makes error of

    'You must feed a value for placeholder tensor 'dense_1_target' with dtype float and shape [?,?]'

    Additionally, when I re-compile the model without any change, it also makes same error at model.fit.

    Is there a way to re-compile?

    I also attach the full error at the end.

    Thanks

    Traceback (most recent call last): File "train_circDrop_declare.py", line 257, in stuffs = model.fit(training_data[tidx][0], training_data[tidx][1], batch_size, epochs_per_task, callbacks=callbacks, verbose=1) File "C:\Users\jjh\Anaconda3\lib\site-packages\keras\engine\training.py", line 1039, in fit validation_steps=validation_steps) File "C:\Users\jjh\Anaconda3\lib\site-packages\keras\engine\training_arrays.py", line 199, in fit_loop outs = f(ins_batch) File "C:\Users\jjh\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in call return self._call(inputs) File "C:\Users\jjh\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "C:\Users\jjh\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1439, in call run_metadata_ptr) File "C:\Users\jjh\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'dense_1_target' with dtype float and shape [?,?] [[{{node dense_1_target}} = Placeholderdtype=DT_FLOAT, shape=[?,?], _device="/job:localhost/replica:0/task:0/device:GPU:0"]] [[{{node loss_1/mul/_417}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3655_loss_1/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

    opened by jjh6297 0
Owner
Ganguli Lab
Ganguli Lab
GitHub
Code and data to accompany the camera-ready version of "Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation" in EMNLP 2021

Code and data to accompany the camera-ready version of "Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation" in EMNLP 2021

Mozhdeh Gheini 16 Jul 16, 2022
Code for the USENIX 2017 paper: kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels

kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels Blazing fast x86-64 VM kernel fuzzing framework with performant VM reloads for Linux, MacOS an

Chair for Sys­tems Se­cu­ri­ty 541 Nov 27, 2022
TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

null 73 Nov 6, 2022
The PyTorch improved version of TPAMI 2017 paper: Face Alignment in Full Pose Range: A 3D Total Solution.

Face Alignment in Full Pose Range: A 3D Total Solution By Jianzhu Guo. [Updates] 2020.8.30: The pre-trained model and code of ECCV-20 are made public

Jianzhu Guo 3.4k Jan 2, 2023
PyTorch implementation of NIPS 2017 paper Dynamic Routing Between Capsules

Dynamic Routing Between Capsules - PyTorch implementation PyTorch implementation of NIPS 2017 paper Dynamic Routing Between Capsules from Sara Sabour,

Adam Bielski 475 Dec 24, 2022
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Karan Desai 105 Nov 25, 2022
PyTorch version of the paper 'Enhanced Deep Residual Networks for Single Image Super-Resolution' (CVPRW 2017)

About PyTorch 1.2.0 Now the master branch supports PyTorch 1.2.0 by default. Due to the serious version problem (especially torch.utils.data.dataloade

Sanghyun Son 2.1k Jan 1, 2023
A PyTorch implementation of the paper "Semantic Image Synthesis via Adversarial Learning" in ICCV 2017

Semantic Image Synthesis via Adversarial Learning This is a PyTorch implementation of the paper Semantic Image Synthesis via Adversarial Learning. Req

Seonghyeon Nam 146 Nov 25, 2022
Implementation supporting the ICCV 2017 paper "GANs for Biological Image Synthesis"

GANs for Biological Image Synthesis This codes implements the ICCV-2017 paper "GANs for Biological Image Synthesis". The paper and its supplementary m

Anton Osokin 95 Nov 25, 2022
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Karan Desai 105 Nov 25, 2022