Implementation of: "Exploring Randomly Wired Neural Networks for Image Recognition"

Overview

RandWireNN

PWC

Unofficial PyTorch Implementation of: Exploring Randomly Wired Neural Networks for Image Recognition.

Results

Validation result on Imagenet(ILSVRC2012) dataset:

Top 1 accuracy (%) Paper Here
RandWire-WS(4, 0.75), C=78 74.7 69.2
  • (2019.06.26) 69.2%: 250 epoch with SGD optimizer, lr 0.1, momentum 0.9, weight decay 5e-5, cosine annealing lr schedule (no label smoothing applied, see loss curve below)
  • (2019.04.14) 62.6%: 396k steps with SGD optimizer, lr 0.1, momentum 0.9, weigth decay 5e-5, lr decay about 0.1 at 300k
  • (2019.04.12) 62.6%: 416k steps with Adabound optimizer, initial lr 0.001(decayed about 0.1 at 300k), final lr 0.1, no weight decay
  • (2019.04) JiaminRen's implementation reached accuarcy which is almost close to paper, using identical training strategy with paper.
  • (2019.04.10) 63.0%: 450k steps with Adam optimizer, initial lr 0.001, lr decay about 0.1 for every 150k step
  • (2019.04.07) 56.8%: Training took about 16 hours on AWS p3.2xlarge(NVIDIA V100). 120k steps were done in total, and Adam optimizer with lr=0.001, batch_size=128 was used with no learning rate decay.

Dependencies

This code was tested on Python 3.6 with PyTorch 1.0.1. Other packages can be installed by:

pip install -r requirements.txt

Generate random DAG

cd model/graphs
python er.py -p 0.2 -o er-02.txt # Erdos-Renyi
python ba.py -m 7 -o ba-7.txt # Barbasi-Albert
python ws.py -k 4 -p 0.75 ws-4-075.txt # Watts-Strogatz
# number of nodes: -n option

All outputs from commands shown above will produce txt file like:

(number of nodes)
(number of edges)
(lines, each line representing edges)

Train RandWireNN

  1. Download ImageNet dataset. Train/val folder should contain list of 1,000 directories, each containing list of images for corresponding category. For validation image files, this script can be useful: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh

  2. Edit config.yaml

    cd config
    cp default.yaml config.yaml
    vim config.yaml # specify data directory, graph txt files
  3. Train

    Note. Validation performed here won't use entire test set, since it will consume much time. (about 3 min.)

    python trainer.py -c [config yaml] -m [name]
    
  4. View tensorboardX

    tensorboard --logdir ./logs
    

Validation

Run full validation:

python validation.py -c [config path] -p [checkpoint path]

This will show accuracy and average test loss of the trained model.

Author

Seungwon Park / @seungwonpark

License

Apache License 2.0

Comments
  • in_degree and out_degree

    in_degree and out_degree

    It is illustrated in figure 2 in origin paper that there are 3 input degrees and 4 output degrees in node operation. In a random graph, we can not guarantee the degree of a node. Is it just a analogy?

    opened by JihaoLee 5
  • How to change the output channels?

    How to change the output channels?

    https://github.com/seungwonpark/RandWireNN/blob/6bea89c423fe7c0d46b37f4256953e905986c84c/model/dag_layer.py#L39 hi, @seungwonpark : When I read your code, I find that you change(i.e., explode) the output channels in the output node instead of the input node. And I review Kaiming He's paper again, there is an explanation "The channel count in a random graph is increased by 2× when going from one stage to the next stage, following [11]", [11] refers to the Residual Network. So I think we should increase the output channels in the input node. How do you think?

    opened by luanyunteng 4
  • Added MNIST and CIFAR10 dataloader funcs

    Added MNIST and CIFAR10 dataloader funcs

    Hello!

    I was trying to test some RandWireNNs but did not have enough compute available to prototype and quickly iterate with different random graphs on ImageNet. I decided to add MNIST and CIFAR10 support to allow for much easier experimentation. I suppose this might come in handy to other people trying to do the same.

    MNIST comes with standard normalization transfs whereas CIFAR10 is raw (directly from https://www.cs.toronto.edu/~kriz/cifar.html). Additionally, the MNIST loader function directly downloads the dataset to ./data folder.

    opened by Zymrael 4
  • How long to train an epoch?

    How long to train an epoch?

    Hi, I am excited to run your code. I run it on 4 TITAN Xp GPU. And yet, each epoch roughly takes 2 hours. Is that normal? What about your time to train an epoch? Thanks!

    opened by Ernie1 3
  • Change train/evaluation process & learning rate scheduler

    Change train/evaluation process & learning rate scheduler

    Currently, training is done iteratively in while loop and is evaluated every 1000 steps. I think we'd better perform the evaluation at the end of every epoch. I will fix this.

    opened by seungwonpark 2
  • Added seamless support for different datasets (trainer.py, validation.py)

    Added seamless support for different datasets (trainer.py, validation.py)

    Hello there!

    I added support for different datasets in trainer.py and validation.py by adaptively creating the correct dataloader based on hp.data.type in default.yaml and test.yaml. To do so I had to slightly change the dataloader funcs themselves so that they accept the same args as the base ImageNet dataloader function.

    I tested the new features on MNIST and as expected it is trivially solved by this model (accuracy not shown here but it was pushing 99% and I only trained for a few thousand iters). The integration is seamless in that nothing is changed in how you call trainer.py from console, although I had to add a new hp.model.input_maps since MNIST images have only 1 input map instead of 3.

    MNIST tloss

    I plan on properly testing CIFAR10 next. I'll send you the plots when that is done. MP

    opened by Zymrael 1
  • Critical: node index assignment of BA, WS should not be completely random

    Critical: node index assignment of BA, WS should not be completely random

    From the appendix of paper,

    The node indexing strategies for the models are — ER: indices are assigned in a random order; BA: the initial M nodes are assigned indices 1 to M, and all other nodes are indexed following their order of adding to the graph; WS: indices are assigned sequentially in the clockwise order.

    The current implementation is assigning random indices for both ER, BA, WS. Since BA/WS generated graphs should not be assigned with random node indices, this is a critical issue. Must be experimented again.

    bug 
    opened by seungwonpark 0
  • NodeOp should not have agg. weight when input_node

    NodeOp should not have agg. weight when input_node

    For input nodes of DAGlayer, NodeOp should not multiply input with learnable parameter - it should be passed by itself. I’ll fix it when I become available. (Currently I’m on vacancy for some reason.)

    opened by seungwonpark 0
  • accuracy

    accuracy

    oh, I try to run your code,but I don't have gpu to run ImageNet,so I runned it using tiny-ImageNet which have 200 classes,but it's accuracy up to 30%, And the test loss raised up. Can you help me

    opened by 1061136002 0
Owner
Seung-won Park
SNU Physics + CSE undergrad., ex-@mindslab-ai
Seung-won Park
ALBERT-pytorch-implementation - ALBERT pytorch implementation

ALBERT-pytorch-implementation developing... 모델의 개념이해를 돕기 위한 구현물로 현재 변수명을 상세히 적었고

BG Kim 3 Oct 6, 2022
Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

NuPIC Numenta Platform for Intelligent Computing The Numenta Platform for Intelligent Computing (NuPIC) is a machine intelligence platform that implem

Numenta 6.3k Dec 30, 2022
PyTorch implementation of neural style transfer algorithm

neural-style-pt This is a PyTorch implementation of the paper A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias

null 770 Jan 2, 2023
PyTorch implementation of DeepDream algorithm

neural-dream This is a PyTorch implementation of DeepDream. The code is based on neural-style-pt. Here we DeepDream a photograph of the Golden Gate Br

null 121 Nov 5, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 5, 2023
Image-to-Image Translation with Conditional Adversarial Networks (Pix2pix) implementation in keras

pix2pix-keras Pix2pix implementation in keras. Original paper: Image-to-Image Translation with Conditional Adversarial Networks (pix2pix) Paper Author

William Falcon 141 Dec 30, 2022
Python implementation of cover trees, near-drop-in replacement for scipy.spatial.kdtree

This is a Python implementation of cover trees, a data structure for finding nearest neighbors in a general metric space (e.g., a 3D box with periodic

Patrick Varilly 28 Nov 25, 2022
Home repository for the Regularized Greedy Forest (RGF) library. It includes original implementation from the paper and multithreaded one written in C++, along with various language-specific wrappers.

Regularized Greedy Forest Regularized Greedy Forest (RGF) is a tree ensemble machine learning method described in this paper. RGF can deliver better r

RGF-team 364 Dec 28, 2022
Implementation of Restricted Boltzmann Machine (RBM) and its variants in Tensorflow

xRBM Library Implementation of Restricted Boltzmann Machine (RBM) and its variants in Tensorflow Installation Using pip: pip install xrbm Examples Tut

Omid Alemi 55 Dec 29, 2022
A fast Evolution Strategy implementation in Python

Evostra: Evolution Strategy for Python Evolution Strategy (ES) is an optimization technique based on ideas of adaptation and evolution. You can learn

Mika 251 Dec 8, 2022
🌳 A Python-inspired implementation of the Optimum-Path Forest classifier.

OPFython: A Python-Inspired Optimum-Path Forest Classifier Welcome to OPFython. Note that this implementation relies purely on the standard LibOPF. Th

Gustavo Rosa 30 Jan 4, 2023
Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

Geometric Vector Perceptron Implementation of Geometric Vector Perceptron, a simple circuit with 3d rotation equivariance for learning over large biom

Phil Wang 59 Nov 24, 2022
Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"

Description: This is the official implementation of our AAAI-21 accepted paper Label Confusion Learning to Enhance Text Classification Models. The str

null 101 Nov 25, 2022
Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

null 49 Nov 23, 2022
PyTorch implementation of "Conformer: Convolution-augmented Transformer for Speech Recognition" (INTERSPEECH 2020)

PyTorch implementation of Conformer: Convolution-augmented Transformer for Speech Recognition. Transformer models are good at capturing content-based

Soohwan Kim 565 Jan 4, 2023
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
The official implementation of NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021]. https://arxiv.org/pdf/2101.12378.pdf

NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021] Release Notes The offical PyTorch implementation of NeMo, p

Angtian Wang 76 Nov 23, 2022
A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

Exploring simple siamese representation learning This is a PyTorch re-implementation of the SimSiam paper on ImageNet dataset. The results match that

Taojiannan Yang 72 Nov 9, 2022
PyTorch implementation of "A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement."

FullSubNet This Git repository for the official PyTorch implementation of "A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech E

郝翔 357 Jan 4, 2023