Efficient Neural Architecture Search (ENAS) in PyTorch
PyTorch implementation of Efficient Neural Architecture Search via Parameters Sharing.
ENAS reduce the computational requirement (GPU-hours) of Neural Architecture Search (NAS) by 1000x via parameter sharing between models that are subgraphs within a large computational graph. SOTA on Penn Treebank language modeling.
**[Caveat] Use official code from the authors: link**
Prerequisites
- Python 3.6+
- PyTorch==0.3.1
- tqdm, scipy, imageio, graphviz, tensorboardX
Usage
Install prerequisites with:
conda install graphviz
pip install -r requirements.txt
To train ENAS to discover a recurrent cell for RNN:
python main.py --network_type rnn --dataset ptb --controller_optim adam --controller_lr 0.00035 \
--shared_optim sgd --shared_lr 20.0 --entropy_coeff 0.0001
python main.py --network_type rnn --dataset wikitext
To train ENAS to discover CNN architecture (in progress):
python main.py --network_type cnn --dataset cifar --controller_optim momentum --controller_lr_cosine=True \
--controller_lr_max 0.05 --controller_lr_min 0.0001 --entropy_coeff 0.1
or you can use your own dataset by placing images like:
data
├── YOUR_TEXT_DATASET
│ ├── test.txt
│ ├── train.txt
│ └── valid.txt
├── YOUR_IMAGE_DATASET
│ ├── test
│ │ ├── xxx.jpg (name doesn't matter)
│ │ ├── yyy.jpg (name doesn't matter)
│ │ └── ...
│ ├── train
│ │ ├── xxx.jpg
│ │ └── ...
│ └── valid
│ ├── xxx.jpg
│ └── ...
├── image.py
└── text.py
To generate gif image of generated samples:
python generate_gif.py --model_name=ptb_2018-02-15_11-20-02 --output=sample.gif
More configurations can be found here.
Results
Efficient Neural Architecture Search (ENAS) is composed of two sets of learnable parameters, controller LSTM θ and the shared parameters ω. These two parameters are alternatively trained and only trained controller is used to derive novel architectures.
1. Discovering Recurrent Cells
Controller LSTM decide 1) what activation function to use and 2) which previous node to connect.
The RNN cell ENAS discovered for Penn Treebank and WikiText-2 dataset:
Best discovered ENAS cell for Penn Treebank at epoch 27:
You can see the details of training (e.g. reward, entropy, loss) with:
tensorboard --logdir=logs --port=6006
2. Discovering Convolutional Neural Networks
Controller LSTM samples 1) what computation operation to use and 2) which previous node to connect.
The CNN network ENAS discovered for CIFAR-10 dataset:
(in progress)
3. Designing Convolutional Cells
(in progress)
Reference
- Neural Architecture Search with Reinforcement Learning
- Neural Optimizer Search with Reinforcement Learning
Author
Taehoon Kim / @carpedm20
48 Sep 27, 2022
90 Dec 8, 2022
121 Dec 17, 2022
172 Dec 12, 2022
111 Dec 8, 2022
76 Jan 7, 2023
13 Nov 19, 2022
556 Jan 4, 2023
616 Jan 6, 2023
520 Dec 30, 2022
6.9k Jan 3, 2023
119 Nov 24, 2022
1.4k Jan 1, 2023
360 Dec 10, 2022
9.2k Jan 2, 2023
0 Jul 13, 2021
8 Nov 21, 2022
157 Dec 11, 2022